Room layout estimation methods and techniques

Abstract

Systems and methods for estimating a room layout are disclosed. The room layout may include the location of a floor, one or more walls, and a ceiling. In one aspect, a neural network may analyze an image of a portion of a room to determine the room layout. The neural network may include a convolutional neural network having an encoder subnetwork, a decoder subnetwork, and a side subnetwork. The neural network may determine a three-dimensional room layout using two-dimensional ordered key points associated with a room type. The room layout may be used for applications such as augmented or mixed reality, robotics, autonomous indoor navigation, and the like.

Description

[0001] This application is a divisional application of the Chinese patent application with application number CN201880017801.5. The application date of the original application is March 16, 2018, the priority date is March 17, 2017, the date of entering the Chinese national phase is September 12, 2019, and the name of the invention is "Room layout estimation method and technology".

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to U.S. Patent Application No. 62 / 473,257, filed on March 17, 2017, entitled “ROOM LAYOUT ESTIMATION METHODS AND TECHNIQUES,” which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates generally to systems and methods for estimating room layout using automatic image analysis, and more particularly to deep machine learning systems (e.g., convolutional neural networks) for determining room layout. Background Art

[0005] A deep neural network (DNN) is a computational machine learning method. DNN belongs to a class of artificial neural networks (NN). Using NN, a computational graph that simulates the characteristics of biological neural networks is constructed. Biological neural networks include features that are prominent for computation and are responsible for many capabilities of biological systems, which otherwise may be difficult to capture by other methods. In some implementations, such a network is arranged into a sequential hierarchical structure with unidirectional connections. For example, the output of an artificial neuron in a particular layer can be connected to the input of an artificial neuron in a subsequent layer. A DNN can be a NN with a large number of layers (e.g., 10, 100, or more layers).

[0006] Different NNs differ from each other in different perspectives. For example, the topology or architecture (e.g., the number of layers and how the layers are interconnected) and weights of different NNs may be different. The weights may be approximately analogous to the synaptic strength of neural connections in biological systems. The weights affect the strength of the effect propagated from one layer to another. The output of an artificial neuron may be a nonlinear function of the weighted sum of its inputs. NNs may be trained on training data and then used to determine outputs from untrained data. Summary of the invention

[0007] Constructing a three-dimensional (3D) representation of the world from images is an important challenge in computer vision and has important applications to augmented reality, robotics, autonomous navigation, etc. The present disclosure provides examples of systems and methods for estimating the layout of a room by analyzing one or more images of the room. The layout may include the position of the floor, one or more walls, the ceiling, etc. in the room.

[0008] In one aspect, a machine learning system including a neural network is used for room layout estimation. In various embodiments, the machine learning system is referred to herein as RoomNet because these various embodiments use a neural network to determine the room layout. The machine learning system can be executed by a hardware computer processor including non-transitory storage, and can be executed locally or in a distributed (e.g., cloud) computing environment.

[0009] The room layout systems and methods described herein are applicable to augmented and mixed reality. For example, an augmented reality (AR) device may include an outward-facing imaging system that is configured to capture images of the environment of the AR device. The AR device may perform a RoomNet analysis of the images to determine the layout of a room in which a wearer of the AR device is located. The AR device may use the room layout to build a 3D representation of the wearer's environment (sometimes referred to as a world map).

[0010] In one aspect, a neural network can analyze an image of a portion of a room to determine a room layout. The neural network can include a convolutional neural network having an encoder subnetwork, a decoder subnetwork, and a side subnetwork. The neural network can determine a three-dimensional room layout using two-dimensional ordered key points associated with a room type. The room layout can be used for applications such as augmented or mixed reality, robotics, autonomous indoor navigation, and the like.

[0011] In one aspect, RoomNet includes an encoder subnetwork, a decoder subnetwork connected to the encoder network, and a side subnetwork connected to the encoder network. After receiving a room image, the encoder subnetwork and the decoder subnetwork of RoomNet can be used to determine multiple predicted heat maps corresponding to multiple room types. The encoder subnetwork and the side subnetwork of RoomNet and the room image can be used to determine the predicted room types for the multiple room types. The predicted heat maps corresponding to the predicted room types can be used to determine key points at multiple predicted key point locations. The predicted room types, key points, and the key point order associated with the predicted room types can be used to determine the predicted layout of the rooms in the room image.

[0012] In another aspect, a system is used to train a neural network for room layout estimation. Training room images may be used to train a neural network, which may include an encoder subnetwork, a decoder subnetwork connected to the encoder network, and a side subnetwork connected to the encoder network. Each of the training room images may be associated with a reference room type and associated with a reference keypoint at a reference keypoint location in the training room image. Training the neural network may include determining a plurality of predicted heatmaps corresponding to the room types using the encoder subnetwork and the decoder subnetwork and the training room images, and determining a predicted room type using the encoder subnetwork and the side subnetwork and the training room images. The neural network may include weights updated based on a first difference between the reference keypoint location and the predicted heatmap and a second difference between the reference room type and the predicted room type.

[0013] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the specification, drawings, and claims. Neither this summary nor the following detailed description is intended to define or restrict the scope of the subject matter of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A is an example pipeline for room layout estimation.

[0015] Figure 1B is an example room layout estimation using an embodiment of a machine learning architecture called RoomNet described herein.

[0016] Figure 1C is another example room layout estimation using RoomNet.

[0017] Figure 2 An example definition of a room layout type is shown. Types can be indexed from 0 to 10. The number on each keypoint defines a specific order of the points saved in the ground truth. For a given room type, the ordering of the keypoints can specify their connectivity.

[0018] Figure 3 Depicts another example architecture of RoomNet.

[0019] Figure 4A An example diagram of an unfolded version of a recurrent neural network (RNN) with three iterations is shown.

[0020] Figure 4B An example RoomNet with a memory-augmented recurrent encoder-decoder (MRED) architecture is shown, which mimics the behavior of RNNs but is designed for static inputs.

[0021] Figures 5A-5D Images illustrating example room layout keypoint estimates from a single image (middle row) without refinement (top row) and with refinement (bottom row) are shown. Keypoint heatmaps from multiple channels are shown in a single two-dimensional (2D) image for visualization purposes.

[0022] Figure 6A-6B Describes the deep supervision without time travel ( Fig. 6A ) and deep supervision through time ( Figure 6B ) in the case of a memory-enhanced recursive encoder-decoder architecture.

[0023] Figures 7A-7G Included are images showing example RoomNet predictions and the corresponding ground truth on the Large Scale Scene Understanding Challenge (LSUN) dataset. RoomNet accesses RGB images as its input (the first column in each figure) and produces example room layout keypoint heatmaps (the second column in each figure). The final keypoints are obtained by extracting the keypoint locations with the maximum response from the heatmap. The third and fourth columns in each figure show the predictions of the RoomNet model by using the same method as in reference. Figure 2 An example quadrilateral room layout representation is generated by connecting the obtained key points in the specific order described. The fifth and sixth columns in each figure show example ground truth.

[0024] Figures 8A-8D Examples of room layout predictions from an embodiment of RoomNet that do not match the (human-annotated) ground truth layout well are shown. The first column in each figure shows an example input image. The second column in each figure shows an example predicted keypoint heatmap. The third and fourth columns in each figure show example quad representations obtained. The fifth and sixth columns show example ground truth.

[0025] Figures 9A-9F Depicts an example encoder-decoder architecture: Fig. 9A )Ordinary encoder-decoder;( Fig. 9B ) stacked encoder-decoder; ( Fig. 9C )Stacked encoder-decoder with skip connection; ( Fig.9D )Encoder-decoder with feedback;( Fig.9E ) a memory-enhanced recursive encoder-decoder; and ( Fig.9F )Memory-augmented recursive encoder-decoder with feedback.

[0026] Fig.10 is a flowchart of an example process of training RoomNet.

[0027] Fig.11 is a flowchart of an example process of room layout estimation using RoomNet.

[0028] Fig.12 An example of a wearable display system in which embodiments of RoomNet may be implemented is schematically shown.

[0029] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure. DETAILED DESCRIPTION

[0030] Overview

[0031] A model (such as a function, algorithm, system, etc.) that represents data relationships and patterns can accept inputs and produce outputs corresponding to the inputs in some way. For example, the model can be implemented as a machine learning method, such as a convolutional neural network (CNN) or a deep neural network (DNN). Deep learning is part of a broader family of machine learning methods based on the idea of ​​learning data representations rather than task-specific algorithms, and shows great promise in solving audio-visual computing problems useful for augmented reality, mixed reality, virtual reality, and machine intelligence. In machine learning, a convolutional neural network (CNN or ConvNet) can include a class of deep feedforward artificial neural networks, and CNNs have been successfully applied to analyze visual images. Machine learning methods include a range of methods that can achieve robust and accurate solutions to a wide variety of problems, including eye image segmentation and eye tracking.

[0032] An example of a neural network for room layout estimation, called RoomNet, is disclosed herein. RoomNet can analyze an image of at least a portion of a room to determine the room layout. The room layout can include a representation of the location of a floor, a wall, or a ceiling in the room. The image can, for example, include a monocular image or a grayscale or color (e.g., red, green, blue (RGB)) image. The image can be one or more frames from a video. Other techniques divide room layout estimation into two subtasks: semantic segmentation of the floor, walls, and ceiling to produce layout hypotheses, followed by an iterative optimization step to rank these hypotheses.

[0033] In contrast to these approaches, RoomNet can formulate the room layout problem as estimating an ordered set of room layout keypoints. Given the locations of these ordered keypoints, the room layout and corresponding segmentation can be fully specified. RoomNet can be an end-to-end trainable encoder-decoder network. The RoomNet machine learning architecture can have better performance (e.g., in terms of computational effort, accuracy, etc.). In some embodiments, RoomNet can have an architecture including recursive computation and memory units to refine keypoint locations with similar or identical parameter capacity.

[0034] Stereo images can provide depth information about the room layout. Estimating the room layout from monocular images (which do not include depth information) is challenging. Room layout estimation from monocular images that aims to depict a two-dimensional (2D) representation (e.g., a quadrilateral representation) of an indoor scene has applications for various computer vision tasks such as indoor navigation, scene reconstruction or rendering, or augmented reality. Figure 1A Conventional room layout techniques are shown that take an image 104, extract image features 108 such as local color, texture, and edge cues in a bottom-up manner, followed by vanishing point detection 112. Conventional methods may include a separate post-processing stage for cleaning feature outliers and generating or ranking a large number of room layout hypotheses 116 with structured support vector machines (SVM) or conditional random fields (CRF). In principle, knowledge of the 2D layout 120a and the vanishing points determined using these methods can be used to obtain (e.g., scaled) a 3D reconstruction of the room layout. However, in practice, these conventional methods are complex and the accuracy of the final layout prediction often depends heavily on the quality of the extracted low-level image features, which themselves are susceptible to local noise, scene clutter, and occlusion. Advantageously, embodiments of the RoomNet of the present disclosure may be less susceptible to local noise, scene clutter, and occlusion. Furthermore, the room layout estimates provided by RoomNet may advantageously have better performance than other methods (e.g., in terms of computational effort, such as 200x or 600x).

[0035] In some embodiments, RoomNet can have better performance than other room layout estimation methods based on convolutional neural networks (CNNs), such as deep neural networks, semantic segmentation, and fully convolutional network (FCN) models that produce information-rich edge maps that replace hand-designed low-level image feature extraction. The predicted edge map generated by such an FCN can then be used to sample vanishing lines for layout hypothesis generation and ranking. For example, FCN can be used to learn semantic surface labels such as left wall, front wall, right wall, ceiling, and floor. The raw per-pixel predictions of the FCN can then be refined using connected components and hole filling techniques, followed by the classic vanishing point / line sampling method to produce the room layout. Compared to this approach, which generates a new set of low-level features and may take 30 seconds or more to process each frame, RoomNet can be a more computationally efficient end-to-end trainable CNN.

[0036] In some embodiments, RoomNet's predictions do not need to be post-processed through a potentially expensive hypothesis testing phase to produce a final layout. RoomNet can perform room layout estimation using a top-down approach and can be directly trained to infer both room layout keypoints (e.g., corners) and room types. Once the room type is inferred or determined and the corresponding ordered set of keypoints is located or determined, the keypoints can be connected in a specific order based on the determined room type to obtain a 2D spatial room layout.

[0037] like Figure 1B and 1C As shown in , the RoomNet architecture can be straightforward and simple. As will be further explained below, RoomNet 124 can take an input image 104 (e.g., of size 320 pixels by 320 pixels), process the image through a convolutional encoder-decoder architecture, extract a set of room layout keypoints 128k1-128k6 from a keypoint heatmap 128 corresponding to a particular room layout, and then (optionally) connect the obtained keypoints in a particular order to provide a room layout 120b. The room layout 120b may include the location or orientation of vertical or horizontal surfaces in the room (such as, for example, a floor 132, a ceiling 134, and a wall 136).

[0038] Optionally, the room layout can be regressed as described below. The room layout 120b can be used in a world map, for example, for augmented reality or indoor autonomous navigation or for scene reconstruction or rendering. Optionally, the room layout can be output as a drawing, architectural drawing, etc. The semantic segmentation of the layout surface can be simply obtained as a result of this connectivity and represented as a semantically segmented room layout image 136. Therefore, RoomNet performs the task of room layout evaluation for key point localization. In some embodiments, RoomNet can be a CNN-based encoder-decoder network. In joint key point regression and room layout type classification, RoomNet can be parameter efficient and effective.

[0039] Example keypoint-based room layout representation

[0040] Embodiments of RoomNet may be effective in room layout estimation. RoomNet may be based on an end-to-end trainable and effectively inferable target output representation. RoomNet may compensate or supplement methods based on assigning geometric context or semantic categories (e.g., floor, wall, or ceiling, etc.) to each pixel in an image, and then obtain room layout key points and boundaries based on pixel-by-pixel labels. Deriving layout key points and boundaries from raw pixel outputs may be non-trivial and less efficient than embodiments of RoomNet. In contrast, RoomNet may be based on a model that directly outputs an ordered set of room layout key point locations, so that both key point-based and pixel-based room layout representations may be obtained efficiently with high accuracy. RoomNet may reduce or eliminate ambiguity in pixel-based representations used by other methods. Therefore, embodiments of RoomNet are able to distinguish between different surface identities (e.g., front wall, side wall, floor, ceiling). For example, RoomNet may correctly distinguish between the front wall category and the right wall category, and thereby output a regular label within the same surface, rather than a mixed label. Therefore, RoomNet may have better overall room layout estimation accuracy and performance.

[0041] In some implementations, you can use Figure 2 The keypoint-based room layout representation shown in is used to train RoomNet. Figure 2 A list of example room types 0-10 204rt0-204rt10 is shown, with their corresponding keypoint definitions labeled 1, 2, 3, 4, 5, 6, 7, and / or 8. The number on each keypoint defines a specific order of points saved in the ground truth. These 11 room layout types can cover most possible situations under typical camera poses and common room layout representations under the Manhattan world assumption, where objects, edges, corners in the image are built on a Cartesian grid, resulting in regularities in the image gradient statistics. In various embodiments, a room type can be represented by a plurality of polygonal regions, each corresponding to, for example, a floor, a ceiling, a right wall, a middle wall, a left wall, etc. The room types can be organized by a set of corner keypoints (e.g., corners corresponding to the intersections of polygonal regions). For example, in room type 204rt5, the left wall is bounded by keypoints 1, 2, 5, and 4; the right wall keypoint is bounded by keypoints 1, 3, 6, and 4; the floor is bounded by keypoints 5, 4, 6; and the ceiling is bounded by keypoints 2, 1, 3. Room types can be semantically segmented to identify floors, walls, and ceilings.

[0042] Once the trained RoomNet predicts the correct keypoint locations with the associated room type, these points can be connected in a specific order to produce a four-square room layout representation. For example, room type 7 204rt7 includes four ordered keypoint locations 208k1-208k4, so that a four-square room layout representation can be constructed by connecting keypoint 1 208k1 with keypoint 2 208k2 and keypoint 3 208k3 with keypoint 4 208k4. The 11 room layouts include one room layout type 204rt0 with eight keypoints, three room layout types 204rt1, 204rt2 and 204rt5 with six keypoints, four room layout types 204rt3, 204rt4, 204rt6 and 204rt7 with four keypoints, and three room layout types 204rt8, 204rt9 and 204rt10 with two keypoints. Room layouts with the same number of keypoints may have the same keypoint connectivity (such as room layout types 3 and 4, 204rt3 and 204rt4) or different keypoint connectivity (such as room layout types 1 and 2, 204rt3 and 204rt4). Although 11 room layout types are used in this example, a different number of room layout types (e.g., 5, 10, 15, 20, or more) or different keypoint connectivity may be used in other implementations. Figure 2 Types of room layouts with different arrangements are shown in .

[0043] Example architecture of RoomNet

[0044] The neural network for room layout estimation of the present disclosure may include a convolutional neural network (CNN) for depicting the room layout structure using two-dimensional (2D) key points. The input of RoomNet may be a monocular image, for example, a single red-green-blue (RGB) image or RGB frame from a video. The output of RoomNet may include a set of 2D key points associated with a specific order having an associated room type.

[0045] Keypoint Estimation In some embodiments, RoomNet may include a base network architecture for keypoint estimation and semantic segmentation of room surfaces (such as roof (or ceiling), left wall, right wall, back wall, floor, etc.). Figure 3An example architecture of RoomNet 300 is depicted. In this example architecture, the decoder upsamples its input using the transferred pooling indices from its encoder to produce a sparse feature map, followed by several convolutional layers with trainable filter banks to densify the feature responses. The final decoder output keypoint heatmaps are fed to a regressor with Euclidean loss. A side head with three fully connected layers is attached to the bottleneck layer and is used to train and predict room type classification labels, which are then used to select an associated set of keypoint heatmaps. As described in reference Figure 4B As described in Figure 5, the full model of RoomNet with a recursive encoder-decoder (center dashed block) further performs keypoint refinement.

[0046] Continue to refer Figure 3 , RoomNet 300 may include an encoder subnetwork 304a and a decoder subnetwork 304b. The encoder subnetwork 304a may map an input image 308 to lower resolution feature maps 312a-312e. The decoder subnetwork 304b may upsample the low resolution encoded feature map 312e to higher resolution maps 316a-316b and heat maps 320r0-320r10 (e.g., having the same or lower resolution than the input image 308) for pixel-by-pixel classification. The dimensions of the input image 308, feature maps 312a-312e, 316a-316b, and heat maps 320r0-320r10 are in Figure 3 . The encoder subnetwork 304a may include multiple convolutional layers and pooling layers 324a-324e. The decoder subnetwork 304b may include multiple convolutional layers and upsampling layers 328a-328c. In some embodiments, the decoder subnetwork 304b may perform nonlinear upsampling using pooling indices calculated in the maximum pooling step or layer of the corresponding encoder subnetwork 304b. For example, the weights of the maximum pooling layer used to generate feature map 312e may be used to upsample feature map 312e. As another example, the weights of the maximum pooling layer used to generate feature map 312c may be used to upsample feature map 316a. Pooling indices may minimize or eliminate the need for learning upsampling. The upsampled maps may be sparse and may be convolved with trainable filters to produce dense feature maps 316a, 316b. Compared to other methods for room layout estimation, the encoder-decoder architecture may provide good performance with competitive inference time and efficient memory usage. The number of heat maps 320r0-320r10 may be the number of defined room types, such as 5, 10, 11, 15 or more. Figure 3The number of keypoints associated with each room type is shown. For example, room type 0 320r0 is associated with eight keypoints. For example, each of the eight keypoints can be identified as the highest peak in each of the eight heatmaps 320r0. Therefore, the number of heatmaps 320r0-320r10 output by RoomNet300 can be the total number of keypoints for different room types. Figure 3 In the example shown in , the number of heat maps 320r0-320r10 is 48.

[0047] The infrastructure of RoomNet 300 can take an image 308 of an indoor scene and directly output a set of 2D room layout keypoints to recover the room layout structure. Each keypoint ground truth can be represented by a 2D Gaussian heatmap centered at the true keypoint location as one of the channels in the output layer. In some embodiments, the keypoint heatmaps 320r0-320r10 in a single 2D image can be color-coded for visualization. The encoder-decoder architecture of RoomNet 300 can process the information flow through the bottleneck layer (e.g., convolution and max pooling layer 324e), forcing the bottleneck layer to implicitly model the relationship between the keypoints encoding the 2D structure of the room layout.

[0048] In some embodiments, the decoder subnetwork 304b of RoomNet 300 may upsample feature maps 312e from the bottleneck layer 324e, whose spatial dimensions are 10×10 to 40×40, instead of Figure 3 320 pixels×320 pixels. This reduction in the dimensionality of the output heatmaps 320r0-320r10 to 40 pixels×40 pixels, compared to the dimensionality of the input image 308, can be empirically determined using a 2D keypoint-based representation that has been proposed for effectively modeling room layouts. In some embodiments, the width and height of the heatmaps 320r0-320r10 can be the same as the width and height of the input image 308, such as 320 pixels×320 pixels. Embodiments of RoomNet 300 with different output dimensions may have similar performance. Due to the high computational cost of convolutions at higher resolutions, using the adjusted decoder subnetwork 304b can advantageously reduce (e.g., significantly reduce) memory usage or time cost during training and testing.

[0049] Expandable to multiple room types The framework or architecture of RoomNet 300 is not limited to one specific room type. Embodiments of RoomNet can be generalized for multiple room types without having to train one network per category. Such embodiments of RoomNet 300 can be efficiently and quickly built from the ground up. Figure 3The RoomNet embodiment 300 shown in FIG. 3 can predict room layout keypoints of associated room types relative to an input image in one forward pass. The number of channels in the output layer 328 c can match the total number of keypoints of all defined room types (e.g., Figure 2 RoomNet 300 may also include a side head or side sub-network 304c having connection layers 332a-332c (e.g., fully connected layers) to a bottleneck layer 324e (e.g., a layer typically used for image classification) to predict the room type. Figure 3 Room type prediction as shown in . The side sub-network may include a classifier network to classify the room type in the room image.

[0050] The training examples or room images can be represented as (I, y, t), where y is a list of ground truth coordinates of k keypoints of room type t with input image I. During the training phase, the loss function L can include a first loss for predicting keypoints and a second loss for predicting room type. The first loss can be a Euclidean loss, which can be used as a cost function for layout keypoint heatmap regression. During training, the second loss can be a cross entropy loss (e.g., logarithmic), which can be used for room type prediction. Given a keypoint heatmap regressor (e.g., output from the decoder subnetwork 304b) and the room type classifier ψ (e.g., output from the fully connected side head layer 304c), the loss function L shown in equation [1] can be optimized (e.g., reduced or minimized).

[0051]

[0052] in indicates whether keypoint k appears in the ground truth room type t, Indicates whether the room type index c is equal to the ground truth room type t, the function G is a Gaussian distribution centered on y, and the weight term is λ. For example, the weight term λ (e.g., 5) can be set by cross-validation. The first term in the loss function compares the predicted heat map 320r0-320r10 with the ground truth heat map synthesized for each key point respectively. The ground truth of each key point heat map can be a 2D Gaussian distribution centered on the true key point position, which has a standard deviation of multiple pixels (e.g., 5 pixels). The second term in the loss function can cause the side head 304c fully connected layer 332a-332c to produce a high confidence value relative to the correct room type classification label.

[0053] A forward pass of RoomNet 300 can be used for all defined room types (e.g. Figure 211) generates 2D room layout key points 320r0-320r10. The 2D room layout key points can be in the form of heat maps, where the final key points can be extracted as the maximum values ​​in these heat maps. In some embodiments, if the key point k exists for the ground truth room type t in the current input image I, the loss function (e.g., the loss function L shown in equation [1]) only penalizes the Euclidean regression error, effectively using the predicted room type index to select a corresponding set of key point heat maps to update the regressor. The same strategy can be applied after training RoomNet 300 (e.g., in the testing phase), so that the predicted room type (e.g., by the side network 304c) is used to select the predicted key point heat map in the final output.

[0054] RoomNet extension for keypoint refinement . Recurrent neural networks (RNNs) and their variants long short-term memory (LSTM) can be effective models when processing sequential data. Even if the input image 308 is static, embodiments of RoomNet 300 can also include recursive structures. For example, RoomNet 300 can include recursive convolutional layers and convolutional LSTM (convLSTM) layers. In some embodiments, the recursive features of RoomNet 300 can be similar to models such as fully convolutional networks (FCNs) with conditional random fields, such as recursive neural networks (CRF-RNNs), iterative error feedback networks, recursive CNNs, stacked encoder-decoders, and recursive encoder-decoder networks. Incorporating time series concepts when modeling static inputs can significantly improve the ability of RoomNet 300 to integrate contextual information and reduce prediction errors in some cases.

[0055] This can be accomplished by enabling the central encoder-decoder component 336 (see e.g. Figure 3 The RoomNet architecture is recursively extended by using the central dashed box in the figure. For example, RoomNet 300 may include a memory-enhanced recursive encoder-decoder (MRED) structure 404b (see Figure 4B ) to mimic the behavior of a typical recurrent neural network 404a (see Figure 4A ) in order to refine the predicted keypoint heatmap by iterating artificial time steps - the artificial time steps (e.g., iterations) are created by the recursive structure.

[0056] Each layer 312c-312e, 316a-316b in the MRED structure 404b may share the same weight matrix over different time steps (eg, iterations), which is used to predict the previous prediction h at time step t-1 from the same layer l. l (t-1) and the current input h at time step t in the previous layer l-1 l-1The input feature map of (t) is convolved (denoted by the * symbol) to produce the output at time step t, as shown in equation [2].

[0057]

[0058] in The input and feed-forward weights of layer l, b l is the bias of layer l, and σ is an activation function, such as the rectified linear unit (ReLU) activation function.

[0059] Figure 4B An example overall process of information flow during forward and backward propagation through depth and time within a recursive encoder-decoder structure is shown. A memory-enhanced recursive encoder-decoder (MRED) architecture 404b includes hidden units 408a, 408b to store previous activations that aid in inference at the current time step. Non-limiting example advantages of using the proposed MRED404b architecture include: (1) iteratively exploiting contextual and structural knowledge between key points through hidden / memory units (e.g., not yet explored in recursive convolutional encoder-decoder structures) or (2) weight sharing of convolutional layers in a recursive encoder-decoder, resulting in a deeper network with a fixed number of parameters.

[0060] like Figures 5A-5D As shown in the bottom row of , after refinement, the heatmap of keypoints is clearer. Figures 5A-5D Images illustrating example room layout keypoint estimation from a single image (middle row, images 504a-504d) without refinement (top row, heatmaps 508a-508d) and with refinement (bottom row, heatmaps 512a-512d) are shown. Keypoint heatmaps from multiple channels are shown in a single two-dimensional (2D) image for visualization purposes. The keypoint refinement step produces a more focused and clearer heatmap and removes false positives, if any. Improvements are made by an embodiment of RoomNet 300 with the MRED architecture 404b (see Figure 5C-5D ).

[0061] Deep Supervision Through Time . When stacked, iterative, or recursive convolutional structures are applied, each layer in the network can receive gradients across more layers and / or time steps, resulting in a model that is more difficult to train. For example, iterative error feedback networks may require multi-stage training, and stacked encoder-decoder structures can use intermediate supervision at the end of each encoder-decoder even when batch normalization is used. Training RoomNet 300 can include injecting supervision at the end of each time step. For example, the same loss function L 604, such as the loss function shown in equation [1], can be applied to all time steps. In Figure 6B The three loss functions L1 604a, L2 604b and L3 604c injected at the end of each time step in can be the same or different. Figure 6A-6B Describes the deep supervision without time travel ( Fig. 6A ) and deep supervision through time ( Figure 6B ). Deep supervision can improve the performance of RoomNet 300 over time.

[0062] Example Training

[0063] Dataset . Embodiments of RoomNet 300 are tested on two challenging benchmark datasets: the Hedau dataset and the Large Scale Scene Understanding Challenge (LSUN) room layout dataset. The Hedau dataset contains 209 training, 53 validation, and 105 test images collected from the Web and LabelMe. The LSUN dataset includes 4000 training, 394 validation, and 1000 test images sampled from the SUN database. All input images are rescaled to 320×320 pixels and are only used to train RoomNet 300 from scratch on the LSUN training set. All experimental results are computed using the LSUN Room Layout Challenge toolkit on the original image scale.

[0064] Implementation details . The input of RoomNet 300 is an RGB image of resolution 320×320 pixels, and the output is a heat map of room layout key points with resolution 40×40 with associated room type classification labels. In other implementations, the image resolution or the heat map resolution can be different. The back propagation through time (BPTT) algorithm is applied to train the model with stochastic gradient descent (SGD) with a batch size of 20, 0.5 packet loss rate, 0.9 momentum, and 0.0005 weight decay. The initial learning rate is 0.00001, and it is dropped by a factor of 5 twice at the 150th and 200th generations, respectively. All variants use the same scheme for a total of 225 generations. The encoder and decoder weights are initialized. Batch normalization and rectified linear unit (ReLU) activation functions are also used after each convolutional layer to improve the training process. Horizontal flipping of the input image is used as data augmentation during training. In some embodiments, RoomNet 300 can be implemented in the open source deep learning framework Caffe.

[0065] The ground truth key point heat map may have zero values ​​(background) for most of its area, and only a small part of it corresponds to a Gaussian distribution (foreground associated with the actual key point location). Therefore, due to the imbalance between the foreground and background distributions, the output of the network may tend to converge to zero. In some embodiments, the gradients are weighted based on the ratio between the foreground and background areas of each key point heat map. The gradients of background pixels are reduced by multiplying them by a factor of 0.2, which makes the training significantly more stable. In some cases, pixels in the background include pixels that are farther (e.g., greater than 5 pixels) from the key point than a threshold distance (e.g., the standard deviation of the Gaussian distribution used to generate the ground truth heat map).

[0066] For an embodiment of RoomNet, training from scratch takes about 40 hours on 4 NVIDIA Titan X GPUs. One forward inference of the full model (3 iterations of RoomNet recursion) took 83 milliseconds on a single GPU. To generate the final test predictions, both the original input and flipped versions of the image were run through the network and the heatmaps were averaged together (an average improvement of 0.12% in keypoint error and 0.15% in pixel error). The keypoint locations were chosen to be the locations with the maximum activation corresponding to the heatmap.

[0067] Example Performance

[0068] In some embodiments, the room layout estimation evaluation metrics may include: pixel error and keypoint error. The pixel error may be the pixel-by-pixel error between the predicted surface label and the ground truth label. The keypoint error may be the average Euclidean distance between the predicted keypoint and the annotated keypoint locations normalized by the image diagonal length.

[0069] Accuracy . The performance of RoomNet 300 on the two datasets is listed in Tables 1 and 2. The previous best method is a two-step framework (per-pixel CNN-based segmentation with a separate hypothesis ranking method). The disclosed RoomNet 300 can significantly improve and outperform previous results in both keypoint error and pixel error, achieving state-of-the-art performance. The side head room type classifier achieves 81.5% accuracy on the LSUN dataset.

[0070] Table 1. Performance of the RoomNet architecture on the Hedau dataset.

[0071] method Pixel error (%) Hedau et al. (2009) 21.20 Del Pero et al. (2012) 16.30 Gupta et al. (2010) 16.20 Zhao et al. (2013) 14.50 Ramalingam et al. (2013) 13.34 Mallya et al. (2015) 12.83 Schwing et al. (2012) 12.8 Del Pero et al. (2013) 12.7 Dasgupta et al. (2016) 9.73 RoomNet recursion 3 iterations 8.36

[0072] Table 2. Performance of the RoomNet architecture on the LSUN dataset.

[0073] method Key point error (%) Pixel error (%) Hedau et al. (2009) 15.48 24.23 Mallya et al. (2015) 11.02 16.71 Dasgupta et al. (2016) 8.20 10.63 RoomNet recursion 3 iterations 6.30 9.86

[0074] Table 3. Runtime evaluation of RoomNet on input size 320 pixels × 320 pixels. Compared with other methods, the RoomNet full model (with 3 iterations) achieves 200 times speedup, and the base RoomNet model (without any iterations) achieves 600 times speedup.

[0075]

[0076] Table 4. The impact of the keypoint refinement step using a memory-augmented recursive encoder-decoder architecture on the LSUN dataset.

[0077] method Key point error (%) Pixel error (%) RoomNet basic 6.95 10.46 RoomNet recursion 2 iterations 6.65 9.97 RoomNet recursion 3 iterations 6.30 9.86

[0078] Table 5. The impact of deep supervision through time on the LSUN dataset for RoomNets with 2 and 3 recursive iterations.

[0079]

[0080] Runtime and complexity . The efficiency evaluation for an input image size of 320×320 is shown in Table 3. Compared with another approach for room layout estimation, the full model (RoomNet recursion 3 iterations) achieves 200x speedup, and the base RoomNet without a recursive structure (RoomNet basic) achieves 600x speedup. The time is two forward passes as described herein. Using any of the proposed RoomNet 300 can provide significant inference time reduction and improved accuracy, as shown in Table 4.

[0081] Example RoomNet Analysis

[0082] Recursion and direct prediction . The effect of each component in the RoomNet architecture was studied using the LSUN dataset. Table 4 shows the effectiveness of extending the RoomNet base architecture to a memory-enhanced recursive encoder-decoder network. It is observed that more iterations lead to a reduction in the error rate of both keypoint error and pixel error: RoomNet 300 with an iterative regression to correct the keypoint positions achieves 6.3% keypoint error and 9.86 pixel error compared to RoomNet 300 without a recursive structure that achieves 6.95% keypoint error and 10.46 pixel error. No further significant performance improvement was observed after three iterations. Without being limited by theory, the improvement can come from the same parameter capacity within the network because the weights of the convolutional layers are shared between iterations.

[0083] The impact of deep supervision through time . When a recursive structure with an encoder-decoder architecture is applied, each layer in the network receives not only the gradient across the depth, but also the time steps across the input and the final objective function during training. The effect of adding an auxiliary loss function at different time steps was determined. Table 5 shows the impact of deep supervision using RoomNet 300 with two or three recursive iterations. By adding auxiliary losses for both cases, both the keypoint error and the pixel error are immediately reduced. In some embodiments, the learning problem with deep supervision through different time steps can be easier. When deep supervision through time is not applied, RoomNet 300 with three iterations performs worse than RoomNet 300 with two iterations. This problem is corrected when deep supervision through time is applied. In some embodiments, with more iterations in the recursive structure, deep supervision through time can be applied to successfully train the architecture.

[0084] Qualitative results The qualitative results of RoomNet 300 are Figures 7A-7G Shown in. Figures 7A-7G are images showing example RoomNet predictions and corresponding ground truth on the Large Scale Scene Understanding Challenge (LSUN) dataset. RoomNet takes RGB images as its input 704a-704g (plotted in the first column in each figure) and produces example room layout keypoint heatmaps 708a-708g (the second column in each figure). The final keypoints are obtained by extracting the locations with the maximum response from the heatmaps. The third and fourth columns in each figure show the keypoints obtained by Figure 2712a-712fg, 716a-716g of example four-square room layout representations 712a-712fg, 716a-716g of key points obtained by connecting them in a specific order in the third column. Different surfaces in the third column are shown with different cross-hatching patterns, which can be generated by the segmentation of the layout to identify ceilings, floors, walls, etc. The RoomNet room layout outputs 712a-712g show the floor, ceiling, and walls with different cross-hatching. In representations 716a-716g, the room layout is superimposed on the corresponding input image 704a-704g. The fifth and sixth columns in each figure show example ground truth 720a-720g, 724a-724g for actual room layouts. The correspondence between the room layouts 712a-712g and 716a-716g (determined by RoomNet) and the actual ground truth layouts 720a-720g and 724a-724g is striking. These example results show that RoomNet is robust to key point occlusion of objects (e.g., tables, chairs, beds, etc.). When the image is clear and the room layout boundaries / corners are not occluded, RoomNet 300 can recover the quadrilateral room layout representation with high accuracy. The RoomNet framework is useful for e.g. Figure 7B , 7C The key point occlusion of the objects shown in 7D, 7F (e.g., table, chair, bed, etc.) is also robust.

[0085] Figures 8A-8D are example images illustrating examples where room layout predictions from an embodiment of RoomNet do not quite match the ground truth layout. As described herein, the differences between the RoomNet predictions and the ground truth can be further reduced or eliminated. The first column in each figure shows example input images 804a-804d. The second column in each figure shows example predicted keypoint heatmaps 808a-808d. The third and fourth columns in each figure show representations 812a-812d, 816a-816d of example squares obtained. Different surfaces in the third column are shown with different cross-hatching patterns, which can result from segmentation of the layout to identify ceilings, floors, walls, etc. The fifth and sixth columns show example ground truth 820a-820d, 824a-824d. When the room layout boundaries are barely visible (e.g., Fig. 8A and Figure 8C ), or when there is more than one plausible room layout for a given image of the scene (e.g. Figure 8B and Fig.8D ), RoomNet 300 can be further improved.

[0086] Example Alternative Encoder-Decoder

[0087] The effect of each component in the proposed architecture is empirically determined using the LSUN dataset. For the studied room layout estimation task, Figures 9A-9F The evaluation of the six alternative encoder-decoder architectures shown in includes: (a) Fig. 9A The conventional encoder / decoder 900a (RoomNet basic) shown in FIG. 900a is a RoomNet basic encoder / decoder shown in FIG. 900b. Fig. 9B The stacked encoder-decoder 900b shown in FIG. 900b; (c) as shown in FIG. Fig. 9C The stacked encoder-decoder 900c with jump connections shown in FIG. 900c; (d) as shown in FIG. Fig.9D The encoder-decoder with feedback 900d shown in FIG. 9; (e) as Fig.9E The memory-enhanced recursive encoder-decoder (RoomNet full) 900e shown in FIG. 900b; and (f) as shown in FIG. Fig.9F Memory-enhanced recursive encoder-decoder with feedback 900f is shown in FIG. For certain tasks, some embodiments of RoomNet 300 may have advantages over other embodiments of RoomNet 300; for example, some embodiments of RoomNet may reduce or eliminate Figures 8A-8D Table 6 shows the performance of different variants on the LSUN dataset.

[0088] Table 6. Evaluation of encoder-decoder (enc-dec) variants on the LSUN dataset. Note that the recursive encoder-decoder uses three iteration time steps.

[0089] Model Key point error (%) Pixel error (%) Ordinary enc-dec (RoomNet basic) 6.95 10.46 Stacked enc-dec 6.82 10.31 Stacked enc-dec with jumper 7.05 10.48 Enc-dec with feedback 6.84 10.10 Recursive enc-dec (RoomNet full) 6.30 9.86 Recursive enc-dec with feedback 6.37 9.88

[0090] Comparison of (a) and (b) configurations 900a, 900b indicates that stacking the encoder-decoder network can further improve the performance by forcing the network to implicitly learn the spatial structure of room layout keypoints by placing constraints on multiple bottleneck layers.

[0091] However, adding skip connections as in (c) configuration 900c did not improve performance on this task under the conditions tested. This may be because the size of the training set (thousands) is not as large as other datasets on which it has been evaluated (millions), so skipping layers is not needed for that particular dataset.

[0092] Adding a feedback loop implemented as a concatenation of previous predictions as input and as new input to the same encoder-decoder network in (d) configuration 900d improves performance. At each iteration, the network has access to the suboptimal predictions so far as well as the original input to help inference at the current time step.

[0093] Making the encoder-decoder recursive with memory cells in (e) configuration 900e to behave as an RNN obtains the lowest keypoint error and pixel error (full RoomNet model). The lateral connections in the recursive encoder-decoder allow the network to carry information forward and help prediction at future time steps. Adding a feedback loop to the memory-enhanced recursive encoder-decoder in (f) configuration 900f does not improve the results. The use of the memory-enhanced structure in configuration (e) 900e may have been good at storing previously hidden state information without feedback. The weight matrices of the encoder-decoder are not shared in (b) and (c) configurations 900b, 900c, but are shared in (d), (e) and (f) configurations 900d, 900e, 900f, resulting in a more efficient architecture in terms of parameters.

[0094] Transferring features through pre-training . To decouple the performance gain due to external data, the results of fine-tuning RoomNet from the SUN pre-trained model (on the semantic segmentation task) were determined. As shown in Table 7, this RoomNet achieved 6.09% keypoint error and 9.04% pixel error, compared to other methods with at least 7.95% keypoint error and 9.31% pixel error on the LSUN dataset. Table 7 reflects the room layout estimation results with additional data or pre-trained models. In some embodiments, RoomNet can be trained using an additional Hedau+ training set and trained from a NYUDv2 RGBD (RGB plus depth) pre-trained model. Table 7 shows the fine-tuning results of RoomNet pre-trained from PASCAL and SUN. The SUN pre-trained RoomNet achieved the lowest keypoint error and pixel error on the LSUN dataset.

[0095] Table 7. Evaluation methods using pre-training techniques on the LSUN dataset.

[0096]

[0097] In some embodiments, RoomNet 300 may include a gating mechanism to allow input signals to change the state of the recurrent unit. In some embodiments, RoomNet 300 may be trained using sequential data and / or a building room layout may be predicted using sequential data.

[0098] Example process of training RoomNet

[0099] Fig.10is a flow chart of an example process 1000 for training RoomNet. Process 1000 may be performed by a hardware processor including a non-transitory memory configured to store images, RoomNet architecture and parameters (e.g., NN weights), room types, 2D keypoint locations (e.g., heatmaps), room layouts, etc.

[0100] Process 1000 begins at block 1004, where training room images for a number of types of rooms and room types are received. Each training room image can be associated with a reference room type and reference key points identifying the room layout (e.g., floor, ceiling, (multiple) walls). In some cases, the training images are annotated by hand to indicate the ground truth of the room shown in the image (e.g., key point locations and room type). The training room images can be monocular images, red-green-blue (RGB) images, etc. The training images can be obtained from the Hedau dataset or the LSUN dataset.

[0101] Process 1000 may include executing a data augmentation strategy (e.g., augmenting the training data with horizontally flipped images) to improve the performance of the trained RoomNet. The number of room types may be different in different implementations, such as 2, 3, 5, 10, 11, 15, 20, or more. The room type may be associated with a plurality of key points associated with a key point sequence. The key points may be connected in the key point sequence to provide a room layout. The number of key points may be different in different implementations, such as 2, 3, 5, 6, 8, 10, 20, 50, or more.

[0102] At block 1008, a neural network (e.g., RoomNet) for room layout estimation may be generated. As described herein, an embodiment of RoomNet may include: an encoder subnetwork, a decoder subnetwork connected to the encoder network, and a side head or subnetwork connected to the encoder network. The encoder subnetwork may include a plurality of convolutional layers and a plurality of pooling layers. The decoder subnetwork may include a plurality of convolutional layers and a plurality of upsampling layers. The weights of the decoder layer of the decoder subnetwork may include the weights of the corresponding encoder layer of the encoder subnetwork. Alternatively or additionally, the weights of the decoder layer of the decoder subnetwork may be the same as the weights of the corresponding encoder layer of the encoder subnetwork. In some embodiments, the encoder subnetwork and the decoder subnetwork include a plurality of recursive layers to form a recursive encoder-decoder structure (e.g., a memory enhanced recursive encoder-decoder (MRED) network). The number of recursive iterations of the recursive layer may be 2, 3, 5, 10, or more. In some embodiments, the weight associated with the first recursive iteration of the iteration of the recursive layer is the same as the weight associated with the second recursive iteration of the current layer.

[0103] The encoder subnetwork and the decoder subnetwork may have different architectures in different implementations. For example, the encoder subnetwork and the decoder subnetwork may have a stacked encoder-decoder architecture. As another example, the encoder subnetwork and the decoder subnetwork may have a stacked encoder-decoder architecture with jump connections. As another example, the encoder subnetwork and the decoder subnetwork may have a stacked encoder-decoder architecture with feedback. In one example, the encoder subnetwork and the decoder subnetwork have a memory-enhanced recursive encoder-decoder (MRED) architecture. In another example, the encoder subnetwork and the decoder subnetwork have a memory-enhanced recursive encoder-decoder (MRED) architecture with feedback. In some embodiments, the feature map of a RoomNet with a recursive layer may be determined using equation [2].

[0104] At block 1012, a plurality of predicted 2D keypoints for each of the room types may be determined using the encoder subnetwork and the decoder subnetwork of RoomNet and the training room images. The dimensions of the training room images may be smaller, the same, or larger than the dimensions of the predicted heatmaps. In some cases, the 2D keypoint locations may be extracted from the heatmaps (e.g., as the maximum values ​​in the heatmaps).

[0105] At block 1016, the predicted room type may be determined using the encoder subnetwork and the side subnetwork of RoomNet and the training room images. For example, the side subnetwork may include multiple layers, such as fully connected layers. In some embodiments, the side subnetwork includes three fully connected layers. The dimension of the output layer of the side subnetwork and the number of multiple room types may be the same.

[0106] At box 1020, process 1000 can optimize (e.g., reduce or minimize) a loss function based on a first loss representing the error in the predicted keypoint relative to the reference keypoint in the training image and a second loss representing the error in the predicted room type relative to the reference room type in the training image. An example of a loss function L is described with reference to equation [1]. The first loss can be a Euclidean loss between the predicted keypoint and the reference keypoint. In some implementations, the predicted keypoint is represented by a heat map, and a reference heat map for the reference (e.g., ground truth) keypoint can be generated by placing a Gaussian distribution at the center of the reference keypoint location. If the keypoint exists for the reference room type in the input training image, the first loss is set to penalize the predicted keypoint. The second loss can be a cross-entropy (e.g., logarithmic) loss based on a room type classifier (e.g., the output from a room type side subnetwork), which encourages the side subnetwork to produce a high confidence value relative to the correct room type.

[0107] In some embodiments, determining the first loss includes determining a reference heat map using the reference key points and determining the difference between the reference heat map and the predicted heat map of the predicted key points. The reference heat map may include a distribution centered on each reference key point position. The distribution may include a two-dimensional Gaussian distribution. The Gaussian distribution may have a standard deviation of, for example, 2, 3, 5, 10 or more pixels. The Gaussian distribution may have a standard deviation of a percentage of the dimension of the reference heat map, such as 5%, 10%, 20%, 25% or more. In some embodiments, determining the reference heat map may include, for example, reducing the value of a pixel by multiplying the pixel value by a degradation factor less than 1 (such as 0.1, 0.2, 0.3), which is a threshold number of pixels away from the reference key point position.

[0108] At block 1024, the neural network parameters of RoomNet may be updated based on the optimized loss function. In some embodiments, the weights of RoomNet may be updated by back propagation.

[0109] Process 1000 can be iterated for each training image in a training image set (e.g., Hedau or LSUN datasets) to adjust the neural network to produce a robust neural network model with reduced or minimized error (e.g., pixel error or keypoint error as described above). Embodiments of RoomNet can be trained using process 1000 and then applied to real-world images in augmented or mixed reality, indoor navigation, scene reconstruction or rendering, etc.

[0110] Example process using RoomNet for room layout estimation

[0111] Fig.11 1 is a flow chart of an example process 1100 for estimating a room layout from a room image using RoomNet. Process 1100 may be performed by a hardware processor including a non-transitory memory configured to store images, RoomNet architecture and parameters (e.g., NN weights), room layout types, heat maps, room layouts, etc. Fig.12 As described, the wearable display system 1200 or the robotic system may be configured to implement an embodiment of process 1100 .

[0112] Process 1100 begins at block 1104, where a system (e.g., reference Fig.12 The wearable display system 1200 described herein receives an input image including a possible room scene. The image may include a room scene. The image may include a color image (e.g., RGB or RGB-D), and the image may be monocular. The image may be a frame of a video, and may be captured using a reference image. Fig.12 The described wearable display system 1200 is obtained by the outward-facing imaging system 1244.

[0113] At block 1108, the wearable display system 1200 may access a neural network for room layout estimation (RoomNet), such as that provided by Fig.10 RoomNet trained by process 1000 shown in . RoomNet may include an encoder subnetwork, a decoder subnetwork connected to the encoder network, and a side subnetwork connected to the encoder network. In some embodiments, the encoder subnetwork and the decoder subnetwork include multiple recursive layers. For example, the encoder-decoder subnetwork may include a memory-enhanced recursive encoder-decoder (MRED) network. The feature map of RoomNet with recursive layers can be determined using equation [2] above. The architecture of RoomNet may be different in different implementations. For example, the encoder subnetwork and the decoder subnetwork may include a stacked encoder-decoder architecture. As another example, the encoder subnetwork and the decoder subnetwork may include multiple jumps. In one example, the side subnetwork includes multiple feedback layers.

[0114] At block 1112, using the encoder subnetwork and decoder subnetwork of RoomNet and the room image, process 1100 can determine a plurality of 2D keypoints corresponding to each of a plurality of room types. The 2D keypoints can be associated with a heat map, and the keypoint locations can be extracted from the heat map as the maximum values ​​occurring in the heat map. The number of room types can be greater than 5 (e.g., 11 in some cases).

[0115] At block 1116, the wearable display system 1200 may determine a predicted room type from a plurality of room types using a side subnetwork of RoomNet and the room image.

[0116] At block 1120, process 1100 may determine a layout of a room in the room image from the predicted room type and the 2D keypoints. The room layout may include ordered keypoints with a keypoint order associated with the predicted room type. The number of keypoints may be different in different implementations, such as 2, 3, 4, 6, 8, 10, or more. The room layout may include semantic segmentation of layout surfaces, such as identification of layout surfaces such as ceilings, floors, walls, etc. The semantic segmentation may be derived from the ordered 2D keypoints. Thus, a neural network may provide a 3D room layout structure using the 2D keypoints, which may be in a specific order with an associated room type.

[0117] At block 1124 , process 1100 may utilize the room layout in an augmented or mixed reality application, for autonomous indoor navigation, for scene reconstruction or rendering, and the like.

[0118] Wearable display system 1200 (see below) Fig.12Description) can interact with a user of the system based on the predicted layout of the room in the input image. In some embodiments, the wearable display system 1200 can perform indoor navigation based on the predicted layout of the room in the room image, such as (for example, by rendering the room layout and (optionally) a path to that location) to guide the user to a desired location in the room. In other embodiments, the wearable display system 1200 can reconstruct the scene in the room image based on the predicted layout of the room in the room image.

[0119] Example NN layer

[0120] As described above, embodiments of RoomNet may include a neural network. A layer of a neural network (NN), such as a deep neural network (DNN), may apply a linear or nonlinear transformation to its input to generate its output. A deep neural network layer may be a normalization layer, a convolutional layer, a softsign layer, a rectified linear layer, a cascade layer, a pooling layer, a recursive layer, a class initial layer, or any combination thereof. A normalization layer may normalize the brightness of its input to generate its output by, for example, Euclidean or L2 normalization. For example, a normalization layer may normalize the brightness of multiple images relative to each other at once to generate multiple normalized images as its output. Non-limiting examples of methods for normalizing brightness include local contrast normalization (LCN) or local response normalization (LRN). Local contrast normalization may nonlinearly normalize the contrast of an image by normalizing a local region of the image on a per-pixel basis to have a mean of zero and a variance of one (or a mean and variance of other values). Local response normalization can normalize the image on a local input region to have a mean of zero and a variance of one (or other values ​​of mean and variance). Normalization layers can speed up the training process.

[0121] A convolutional layer may apply a set of kernels that convolve its input to generate its output. A softsign layer may apply a softsign function to its input. The softsign function (softsign(x)) may be, for example, (x / (1+|x|)). The softsign layer may ignore the effects of outliers for each element. A rectified linear layer may be a rectified linear layer unit (ReLU) or a parameterized rectified linear layer unit (PReLU). A ReLU layer may apply a ReLU function to its input to generate its output. The ReLU function ReLU(x) may be, for example, max(0, x). A PReLU layer may apply a PReLU function to its input to generate its output. For example, if x≥0, the PReLU function PReLU(x) may be x, and if x<0, it may be ax, where a is a positive number. A cascade layer may cascade its input to generate its output. For example, a cascade layer may cascade four 5×5 images to generate a 20×20 image. A pooling layer may apply a pooling function that downsamples its input to generate its output. For example, a pooling layer may downsample a 20×20 image to a 10×10 image. Non-limiting examples of pooling functions include max pooling, average pooling, or min pooling.

[0122] At time point t, the recursive layer can calculate a hidden state s(t), and the recursive connection can provide the hidden state s(t) to the recursive layer at time t as an input at a subsequent time point t+1. The recursive layer can calculate its output at time t+1 based on the hidden state s(t) at time t. For example, the recursive layer can apply a softsign function to the hidden state s(t) at time t to calculate its output at time t+1. The hidden state of the recursive layer at time t+1 has the hidden state s(t) of the recursive layer at time t as its input. The recursive layer can calculate the hidden state s(t+1) by applying, for example, a ReLU function to its input. The class initiation layer may include one or more of a normalization layer, a convolutional layer, a softsign layer, a rectified linear layer (such as a ReLU layer and a PReLU layer), a cascade layer, a pooling layer, or any combination thereof.

[0123] In different implementations, the number of layers in the NN can be different. For example, the number of layers in the DNN can be 50, 100, 200, or more. The input types of the deep neural network layers can be different in different implementations. For example, a layer can receive the outputs of multiple layers as its input. The input of a layer can include the outputs of five layers. As another example, the input of a layer can include 1% of the layers of the NN. The output of a layer can be the input of multiple layers. For example, the output of a layer can be used as the input of five layers. As another example, the output of a layer can be used as the input of 1% of the layers of the NN.

[0124] The input size or output size of a layer can be very large. The input size or output size of a layer can be n×m, where n represents the width of the input or output, and m represents the height of the input or output. For example, n or m can be 11, 21, 31, or more. In different implementations, the channel size of the input or output of a layer can be different. For example, the channel size of the input or output of a layer can be 4, 16, 32, 64, 128, or more. In different implementations, the kernel size of a layer can be different. For example, the kernel size can be n×m, where n represents the width of the kernel, and m represents the height of the kernel. For example, n or m can be 5, 7, 9, or more. In different implementations, the stride size of a layer can be different. For example, the stride size of a deep neural network layer can be 3, 5, 7, or more.

[0125] In some embodiments, NN may refer to multiple NNs that together compute the output of the NN. Different NNs in the multiple NNs may be trained for different tasks. A processor (e.g., referring to Fig.12 The processor of the local data processing module 1224 described in the foregoing description may calculate the output of the NN of the plurality of NNs to determine the output of the NN. For example, the output of the NN of the plurality of NNs may include a likelihood score. The processor may determine the output of the NN including the plurality of NNs based on the likelihood scores of the outputs of different NNs of the plurality of NNs.

[0126] Example Wearable Display System

[0127] In some embodiments, the user device may be or may be included in a wearable display device that may advantageously provide a more immersive virtual reality (VR), augmented reality (AR), or mixed reality (MR) experience in which digitally reproduced images, or portions thereof, are presented to the wearer in a manner that they appear to be or may be perceived as real.

[0128] Without being limited by theory, it is believed that the human eye can generally interpret a finite number of depth planes to provide depth perception. Therefore, by providing the eye with a different presentation of an image corresponding to each of these finite number of depth planes, a highly convincing simulation of perceived depth can be achieved. For example, a display including a waveguide stack can be configured to be worn and positioned in front of the eyes of a user or viewer. A waveguide stack can be utilized to provide a three-dimensional perception to the eye / brain by using multiple waveguides to direct light from an image injection device (e.g., a discrete display or output of a multiplexed display that transmits image information via one or more optical fibers) to the viewer's eyes at specific angles (and divergences) corresponding to the depth planes associated with a particular waveguide.

[0129] In some embodiments, two waveguide stacks may be utilized, one for each eye of the viewer, to provide a different image to each eye. As an example, an augmented reality scene may cause the wearer of the AR technology to see a real-world park-like setting featuring people, trees, buildings in the background, and a specific platform. In addition to these items, the wearer of the AR technology may also perceive that he "sees" a robot statue standing on the real-world platform, and a cartoon-like avatar character that appears to be an anthropomorphic representation of a bumblebee flying by, even though the robot statue and the bumblebee do not exist in the real world. (Multiple) waveguide stacks may be used to generate a light field corresponding to an input image, and in some implementations, the wearable display includes a wearable light field display. Examples of wearable display devices and waveguide stacks for providing light field images are described in U.S. Patent Publication No. 2015 / 0016777, the entire contents of which are incorporated herein by reference in their entirety.

[0130] Fig.12 An example of a wearable display system 1200 is shown that can be used to present a VR, AR, or MR experience to a display system wearer or viewer 1204. The wearable display system 1200 can be programmed to perform any application or embodiment described herein (e.g., estimating a room layout using RoomNet). The display system 1200 includes a display 1208, and various mechanical and electronic modules and systems that support the functionality of the display 1208. The display 1208 can be coupled to a frame 1212 that can be worn by the display system wearer or viewer 1204 and is configured to position the display 1208 in front of the eyes of the wearer 1204. The display 1208 can be a light field display. In some embodiments, a speaker 1216 is coupled to the frame 1212 and in some embodiments is positioned adjacent to the user's ear canal, and another speaker (not shown) is positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control. The display system 1200 may include an outward-facing imaging system 1244 (e.g., one or more cameras) that may obtain images (e.g., still images or video) of the environment surrounding the wearer 1204. The images obtained by the outward-facing imaging system 1244 may be analyzed by an embodiment of RoomNet to determine a representation of a room layout in the environment surrounding the wearer 1204.

[0131] The display 1208 is operably coupled 1220 to a local data processing module 1224, such as by a wired lead or a wireless connection, which can be mounted in various configurations, such as fixedly attached to the frame 1212, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or otherwise removably attached to the user 1204 (e.g., in a backpack configuration, in a belt-coupled configuration).

[0132] The local processing and data module 1224 may include a hardware processor, as well as non-transitory digital memory, such as non-volatile memory, e.g., flash memory, both of which may be used to assist in the processing, caching, and storage of data. The data may include: (a) data captured from sensors such as image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes (which may, for example, be operably coupled to the frame 1212 or otherwise attached to the wearer 1204); and / or (b) data acquired and / or processed using the remote processing module 1228 and / or remote data repository 1232, possibly for display 1208 after such processing or acquisition. The local processing and data module 1224 may be operably coupled to the remote processing module 1228 and the remote data repository 1232 by communication links 1236, 1240, such as via a wired or wireless communication link, such that these remote modules 1228, 1232 are operably coupled to each other and may serve as resources for the local processing and data module 1224. Image capture device(s) may be used to capture eye images used in eye image segmentation or eye tracking processes.

[0133] In some embodiments, the remote processing module 1228 may include one or more processors configured to analyze and process data and / or image information, such as video information captured by an image capture device (e.g., by executing RoomNet). Video data may be stored locally in the local processing and data module 1224 and / or in the remote data repository 1232. In some embodiments, the remote data repository 1232 may include a digital data storage facility that may be available via the Internet or other network configuration in a "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 1224 allowing for full autonomous use from the remote module.

[0134] In some implementations, the local processing and data module 1224 and / or the remote processing module 1228 are programmed to execute an embodiment of RoomNet to determine a room layout. For example, the local processing and data module 1224 and / or the remote processing module 1228 may be programmed to execute a reference Fig.11Embodiments of the process 1100 described herein. The local processing and data module 1224 and / or the remote processing module 1228 can be programmed to perform the room layout estimation method 1100 disclosed herein. An image capture device can capture video for a particular application (e.g., typically augmented reality (AR) or mixed reality (MR), human-computer interaction (HCI), autonomous vehicles, drones, or robots). The video (or one or more frames from the video) can be analyzed by one or both of the processing modules 1224, 1228 using an embodiment of the computational RoomNet architecture. In some cases, offloading at least some of the RoomNet analysis to a remote processing module (e.g., in the "cloud") can improve the efficiency or speed of the computation. The parameters of the RoomNet neural network (e.g., weights, bias terms, subsampling factors for pooling layers, the number and size of kernels in different layers, the number of feature maps, room layout types, key point heat maps, etc.) can be stored in the data module 1224 and / or 1232.

[0135] The results of the RoomNet analysis (e.g., the output of method 1100) can be used by one or both of the processing modules 1224, 1228 for additional operations or processing. For example, the processing modules 1224, 1228 of the wearable display system 1200 can be programmed to perform additional applications such as augmented or mixed reality, indoor navigation, or scene reconstruction or rendering based on the output of method 1100. Thus, the wearable system 200 can use RoomNet to provide room layout in real time.

[0136] For example, the wearable display system 1200 can utilize a world map (e.g., stored in a local or remote data repository 1224, 1240) that describes where objects, walls, floors, ceilings, doors, etc. are located relative to each other in a mixed reality environment. Further details regarding the use of a world map are described in U.S. Patent Publication No. 2015 / 0016777, the entire contents of which are incorporated herein by reference. The output of RoomNet (e.g., the output of method 1100) can be used to update the world map to include the room layout of the room in which the wearer of the system 1200 is located.

[0137] The RoomNet architecture can be used with other object recognizers or deep learning systems that analyze images of objects in the user's environment. For example, U.S. Patent Application No. 15 / 812,928, entitled "Deep Learning System for Cuboid Detection," filed on November 14, 2017, describes machine learning techniques for detecting 3D cuboid-shaped objects in images, the entire contents of which are incorporated herein by reference. In some embodiments, the RoomNet architecture can be used to recognize room layouts, and the cuboid detection architecture can be used to recognize or locate cubic objects within the room layout. This information can be added to the world map of the wearable display system 1200 to provide an improved AR or MR user experience.

[0138] As another example, a robot may utilize an embodiment of RoomNet to determine a room layout, and then use the room layout for autonomous navigation of the robot within the room. The robots may include autonomous indoor robots (e.g., robotic vacuums, mops, sweepers), warehouse robots (e.g., for automated storage, retrieval, and inventory operations), indoor aerial drones, and the like.

[0139] Additional aspects

[0140] In a first aspect, a system for estimating a room layout is disclosed. The system includes: a non-transitory memory configured to store: a room image for room layout estimation; and a neural network for estimating the room layout, the neural network including: an encoder-decoder subnetwork; and a classifier subnetwork connected to the encoder-decoder subnetwork; a hardware processor in communication with the non-transitory memory, the hardware processor being programmed to: access the room image; determine a plurality of predicted two-dimensional (2D) keypoint maps corresponding to a plurality of room types using the encoder-decoder subnetwork and the room image; determine a predicted room type from the plurality of room types using the encoder-decoder subnetwork, the classifier subnetwork, and the room image; determine a plurality of ordered keypoints associated with the predicted room type using the plurality of predicted 2D keypoint maps and the predicted room type; and determine a predicted layout of a room in the room image using the plurality of ordered keypoints.

[0141] In a second aspect, the system of aspect 1, wherein each room type of the plurality of room types comprises an ordered set of room type key points.

[0142] In a third aspect, the system of aspect 2, wherein each room type of the plurality of room types comprises a semantic segmentation of a region in the room type, the semantic segmentation comprising an identification as a floor, a ceiling, or a wall.

[0143] In aspect 4, a system according to aspect 2 or aspect 3, wherein a first key point sequence is associated with a first room type among a plurality of room types, and a second key point sequence is associated with a second room type among a plurality of room types, wherein the first key point sequence and the second key point sequence are different.

[0144] In a fifth aspect, the system according to any one of aspects 1 to 4, wherein the room image comprises a monocular image.

[0145] In a 6th aspect, according to any one of aspects 1 to 5, the room image comprises a red green blue (RGB) image.

[0146] In a 7th aspect, according to the system of any one of aspects 1 to 6, the dimension of the room image is larger than the dimension of the predicted 2D keypoint map.

[0147] In an eighth aspect, the system according to any one of aspects 1 to 7, wherein the encoder-decoder subnetwork comprises an encoder subnetwork comprising a plurality of convolutional layers and a plurality of pooling layers.

[0148] In a 9th aspect, according to the system of any one of aspects 1 to 8, the encoder-decoder sub-network includes a decoder sub-network, and the decoder sub-network includes multiple convolutional layers and multiple upsampling layers.

[0149] In a 10th aspect, the system according to any one of aspects 1 to 9, wherein the encoder-decoder sub-network comprises a memory enhanced recursive encoder-decoder (MRED) network.

[0150] In an 11th aspect, the system according to any one of aspects 1 to 10, wherein the encoder-decoder sub-network comprises a plurality of recursive layers.

[0151] In a 12th aspect, according to the system of aspect 11, the number of recursive iterations of the plurality of recursive layers is two.

[0152] In a 13th aspect, the system of aspect 11, wherein the number of recursive iterations of the plurality of recursive layers is at least three.

[0153] In a 14th aspect, the system according to any one of aspects 11 to 13, wherein each of the plurality of recursive layers has a weight matrix, and the weight matrix is ​​the same for all recursive layers in the plurality of recursive layers.

[0154] In a 15th aspect, the system according to any one of aspects 1 to 14, wherein the predicted two-dimensional (2D) keypoint map comprises a heat map.

[0155] In a 16th aspect, the system of aspect 15, wherein the hardware processor is programmed to extract the keypoint location from the heat map as a maximum value of the heat map.

[0156] In a 17th aspect, the system of any of aspects 1 to 16, wherein the hardware processor is programmed to: access object information from an object identifier that analyzes the room image; and combine the object information with the predicted room layout.

[0157] In an 18th aspect, the system of aspect 17, wherein the object identifier is configured to detect a cuboid in the image of the room.

[0158] In a 19th aspect, a wearable display device is disclosed. The wearable display device comprises: an outward-facing imaging system configured to capture room images for room layout estimation; and the system of any one of aspects 1 to 18.

[0159] In the 20th aspect, a system for training a neural network for estimating a room layout. The system includes: a non-transitory memory configured to store parameters of the neural network; and a hardware processor in communication with the non-transitory memory, the hardware processor being programmed to: receive a training room image, wherein the training room image is associated with: a reference room type from a plurality of room types, and a reference key point associated with the reference room layout; generate a neural network for room layout estimation, wherein the neural network includes: an encoder-decoder subnetwork configured to output predicted two-dimensional (2D) key points associated with a predicted room layout associated with each of the plurality of room types, and a side subnetwork connected to the encoder-decoder network configured to output predicted room types from the plurality of room types; and optimize a loss function based on a first loss for the predicted 2D key points and a second loss for the predicted room type; and update the parameters of the neural network based on the optimized loss function.

[0160] In a 21st aspect, the system of aspect 20, wherein the number of the plurality of room types is greater than five.

[0161] In aspect 22, the system according to aspect 20 or aspect 21, wherein the reference keypoints and the predicted 2D keypoints are associated with a keypoint sequence.

[0162] In aspect 23, a system according to any one of aspects 20 to 22, wherein a first key point sequence is associated with a first room type of a plurality of room types, and a second key point sequence is associated with a second room type of the plurality of room types, wherein the first key point sequence and the second key point sequence are different.

[0163] In a 24th aspect, the system of any one of aspects 20 to 23, wherein the training room images comprise monocular images.

[0164] In aspect 25, the system of any one of aspects 20 to 24, wherein the training room image comprises a red green blue (RGB) image.

[0165] In aspect 26, the system of any one of aspects 20 to 25, wherein the dimension of the training room images is larger than the dimension of the map associated with the predicted 2D keypoints.

[0166] In aspect 27, the system of any one of aspects 20 to 26, wherein the encoder sub-network and the decoder sub-network comprise a plurality of recursive layers.

[0167] In aspect 28, the system of aspect 27, wherein the number of recursive iterations of the recursive layer is two or three.

[0168] In aspect 29, a system according to aspect 27 or aspect 28, wherein deep supervision is applied to the recursive layer.

[0169] In aspect 30, the system of any of aspects 27 to 29, wherein the weight associated with a first recursive iteration of an iteration of a recursive layer is the same as the weight associated with a second recursive iteration of the current layer.

[0170] In aspect 31, the system of any one of aspects 27 to 30, wherein the plurality of recursive layers are configured as a memory enhanced recursive encoder-decoder (MRED) network.

[0171] In aspect 32, the system of any one of aspects 20 to 31, wherein the side sub-network comprises a room type classifier.

[0172] In aspect 33, the system of any one of aspects 20 to 32, wherein the first loss for the predicted 2D keypoint comprises a Euclidean loss between a plurality of reference keypoint positions and the predicted 2D keypoint.

[0173] In a 34th aspect, the system of any one of aspects 20 to 33, wherein the second loss for predicting the room type comprises a cross entropy loss.

[0174] In aspect 35, the system of any one of aspects 20 to 34, wherein the predicted 2D keypoints are extracted from the predicted heatmap.

[0175] In a 36th aspect, the system of aspect 35, wherein the hardware processor is programmed to: calculate a reference heat map associated with reference key points of the training image; and calculate a first loss for the predicted 2D key points based on a difference between the predicted heat map and the reference heat map.

[0176] In aspect 37, the system of aspect 36, wherein the reference heat map comprises a two-dimensional distribution centered at the location of each reference keypoint.

[0177] In aspect 38, a system according to aspect 36 or aspect 37, wherein the reference heat map includes a background away from the reference key point and a foreground associated with the reference key point, and the hardware processor is programmed to weight the gradient in the reference heat map based on a ratio between the foreground and the background.

[0178] In a 39th aspect, the system of aspect 38, wherein, to weight the gradients in the reference heat map, the hardware processor is programmed to reduce pixel values ​​in the background by a degradation factor less than 1.

[0179] In a 40th aspect, a wearable display system is disclosed. The system includes: an outward-facing imaging system configured to obtain a room image of an environment of a wearer of the wearable display system; a non-transitory memory configured to store the room image; and a hardware processor in communication with the non-transitory memory, the processor being programmed to: access the room image of the environment; and analyze the room image to determine a predicted layout of a room in the room image, wherein to analyze the room image, the processor is programmed to: use a neural network to determine an ordered set of two-dimensional (2D) key points associated with a room type of the room in the room image; and provide a room layout based at least in part on the 2D key points and the room type.

[0180] In a 41st aspect, the wearable display system of aspect 40, wherein the neural network comprises a convolutional encoder-decoder network.

[0181] In a 42nd aspect, the wearable display system of aspect 41, wherein the convolutional encoder-decoder network comprises a memory-augmented recursive encoder-decoder network.

[0182] In a 43rd aspect, the wearable display system of any one of aspects 40 to 42, wherein the neural network comprises a classifier configured to determine the room type.

[0183] In a 44th aspect, the wearable display system of any one of aspects 40 to 43, wherein the hardware processor is further programmed to extract an ordered set of 2D key points from the heat map.

[0184] In aspect 45, the wearable display system of any one of aspects 40 to 44, wherein the hardware processor is further programmed to: access object information from an object identifier that analyzes the room image; and combine the object information with the room layout.

[0185] In a 46th aspect, a method for estimating a room layout is disclosed. The method includes: accessing a room image for room layout estimation; determining a plurality of predicted two-dimensional (2D) keypoint maps corresponding to a plurality of room types using an encoder-decoder subnetwork of a neural network for estimating the room layout and the room image; determining a predicted room type from the plurality of room types using the encoder-decoder subnetwork, a classifier subnetwork of the neural network connected to the encoder-decoder subnetwork, and the room image; determining a plurality of ordered keypoints associated with the predicted room type using the plurality of predicted 2D keypoint maps and the predicted room type; and determining a predicted layout of a room in the room image using the plurality of ordered keypoints.

[0186] In a 47th aspect, the method of aspect 46, wherein each room type of the plurality of room types comprises an ordered set of room type key points.

[0187] In aspect 48, the method of aspect 47, wherein each room type of the plurality of room types comprises a semantic segmentation of a region in the room type, the semantic segmentation comprising an identification as a floor, a ceiling, or a wall.

[0188] In aspect 49, a method according to aspect 47 or aspect 48, wherein the first key point sequence is associated with a first room type of the plurality of room types, and the second key point sequence is associated with a second room type of the plurality of room types, wherein the first key point sequence and the second key point sequence are different.

[0189] In aspect 50, the method of any one of aspects 46 to 49, wherein the room image comprises a monocular image.

[0190] In aspect 51, the method of any one of aspects 46 to 50, wherein the room image comprises a red green blue (RGB) image.

[0191] In aspect 52, a method according to any one of aspects 46 to 51, wherein the dimension of the room image is larger than the dimension of the predicted 2D keypoint map.

[0192] In aspect 53, the method according to any one of aspects 46 to 52, wherein the encoder-decoder sub-network includes an encoder sub-network, which includes a plurality of convolutional layers and a plurality of pooling layers.

[0193] In aspect 54, the method of any one of aspects 46 to 53, wherein the encoder-decoder subnetwork comprises a decoder subnetwork comprising a plurality of convolutional layers and a plurality of upsampling layers.

[0194] In aspect 55, a method according to any one of aspects 46 to 54, wherein the encoder-decoder sub-network comprises a memory enhanced recursive encoder-decoder (MRED) network.

[0195] In aspect 56, a method according to any one of aspects 46 to 55, wherein the encoder-decoder sub-network comprises a plurality of recursive layers.

[0196] In aspect 57, the method of aspect 56, wherein the number of recursive iterations of the plurality of recursive layers is two.

[0197] In aspect 58, the method of aspect 56, wherein the number of recursive iterations of the plurality of recursive layers is at least three.

[0198] In aspect 59, the method of any one of aspects 56 to 58, wherein each of the plurality of recursive layers has a weight matrix, and the weight matrix is ​​the same for all recursive layers in the plurality of recursive layers.

[0199] In aspect 60, the method of any one of aspects 46 to 59, wherein the predicted two-dimensional (2D) keypoint map comprises a heat map.

[0200] In aspect 61, the method according to aspect 60 further comprises extracting the key point location from the heat map as the maximum value of the heat map.

[0201] In aspect 62, the method according to any one of aspects 46 to 61 further comprises: accessing object information from an object identifier analyzing the room image; and combining the object information with the predicted room layout.

[0202] In a 63rd aspect, the method according to aspect 62 further comprises: using an object identifier to detect a cuboid in the room image.

[0203] In a 64th aspect, a method for training a neural network for estimating a room layout is disclosed. The method includes: receiving a training room image, wherein the training room image is associated with: a reference room type from a plurality of room types, and reference key points associated with the reference room layout; generating a neural network for room layout estimation, wherein the neural network includes: an encoder-decoder subnetwork configured to output predicted two-dimensional (2D) key points associated with a predicted room layout associated with each of the plurality of room types, and a side subnetwork connected to the encoder-decoder network configured to output predicted room types from the plurality of room types; and optimizing a loss function based on a first loss for the predicted 2D key points and a second loss for the predicted room type; and updating parameters of the neural network based on the optimized loss function.

[0204] In aspect 65, the method of aspect 64, wherein the number of the plurality of room types is greater than five.

[0205] In aspect 66, the method of aspect 64 or aspect 65, wherein the reference keypoints and the predicted 2D keypoints are associated with a keypoint sequence.

[0206] In aspect 67, a method according to any one of aspects 64 to 66, wherein a first key point sequence is associated with a first room type of a plurality of room types, and a second key point sequence is associated with a second room type of the plurality of room types, wherein the first key point sequence and the second key point sequence are different.

[0207] In aspect 68, the method of any one of aspects 64 to 67, wherein the training room images comprise monocular images.

[0208] In aspect 69, the method of any one of aspects 64 to 68, wherein the training room images comprise red green blue (RGB) images.

[0209] In aspect 70, the method of any one of aspects 64 to 69, wherein the dimension of the training room images is larger than the dimension of the map associated with the predicted 2D keypoints.

[0210] In aspect 71, the method of any one of aspects 64 to 70, wherein the encoder sub-network and the decoder sub-network comprise a plurality of recursive layers.

[0211] In aspect 72, the method of aspect 71, wherein the number of recursive iterations of the recursive layer is two or three.

[0212] In aspect 73, a method according to aspect 71 or aspect 72, wherein deep supervision is applied to the recursive layer.

[0213] In aspect 74, a method according to any one of aspects 71 to 73, wherein the weight associated with a first recursive iteration of an iteration of a recursive layer is the same as the weight associated with a second recursive iteration of the current layer.

[0214] In aspect 75, the method of any one of aspects 71 to 74, wherein the plurality of recursive layers are configured as a memory enhanced recursive encoder-decoder (MRED) network.

[0215] In aspect 76, the method of any one of aspects 64 to 75, wherein the side sub-network comprises a room type classifier.

[0216] In aspect 77, the method of any one of aspects 64 to 76, wherein the first loss for the predicted 2D keypoint comprises a Euclidean loss between a plurality of reference keypoint positions and the predicted 2D keypoint.

[0217] In aspect 78, the method of any one of aspects 64 to 77, wherein the second loss for predicting the room type comprises a cross entropy loss.

[0218] In aspect 79, the method of any one of aspects 64 to 78, wherein the predicted 2D keypoints are extracted from the predicted heatmap.

[0219] In aspect 80, the method according to aspect 79 further comprises: calculating a reference heat map associated with reference key points of the training image; and calculating a first loss for the predicted 2D key points based on the difference between the predicted heat map and the reference heat map.

[0220] In aspect 81, the method of aspect 80, wherein the reference heat map comprises a two-dimensional distribution centered at the location of each reference keypoint.

[0221] In aspect 82, a method according to aspect 80 or aspect 81, wherein the reference heat map includes a background away from the reference key point and a foreground associated with the reference key point, and the hardware processor is programmed to weight the gradient in the reference heat map based on a ratio between the foreground and the background.

[0222] In aspect 83, the method of aspect 82, wherein weighting the gradients in the reference heat map comprises reducing pixel values ​​in the background by a degradation factor less than 1.

[0223] In an 84th aspect, a method is disclosed. The method includes: accessing a room image of an environment; analyzing the room image to determine a predicted layout of rooms in the room image, including: using a neural network to determine an ordered set of two-dimensional (2D) keypoints associated with room types of the rooms in the room image; and providing the room layout based at least in part on the 2D keypoints and the room types.

[0224] In aspect 85, the method of aspect 84, wherein the neural network comprises a convolutional encoder-decoder network.

[0225] In aspect 86, the method of aspect 85, wherein the convolutional encoder-decoder network comprises a memory-augmented recursive encoder-decoder network.

[0226] In aspect 87, the method of any one of aspects 84 to 86, wherein the neural network comprises a classifier configured to determine the room type.

[0227] In aspect 88, the method according to any one of aspects 84 to 87 further comprises extracting an ordered set of 2D keypoints from the heat map.

[0228] In aspect 89, the method according to any one of aspects 84 to 88 further comprises: analyzing the room image to determine object information; and combining the object information with the room layout.

[0229] Additional considerations

[0230] Each of the processes, methods, and algorithms described herein and / or depicted in the accompanying drawings may be embodied in a code module executed by one or more physical computing systems, hardware computer processors, dedicated circuits, and / or electronic hardware configured to execute dedicated and specific computer instructions, and fully or partially automated by the code module. For example, a computing system may include a general-purpose computer (e.g., a server) programmed with specific computer instructions or a dedicated computer, dedicated circuit, etc. The code module may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted programming language. In some implementations, specific operations and methods may be performed by circuits specific to a given function.

[0231] Furthermore, certain implementations of the functionality of the present disclosure are sufficiently mathematically, computationally, or technically complex that dedicated hardware or one or more physical computing devices (utilizing appropriate dedicated executable instructions) may be necessary, for example, due to the amount or complexity of the computations involved to perform the functionality or provide results in substantially real time. For example, a video may include many frames, each with millions of pixels, and specially programmed computer hardware may be required to process the video data to provide a desired image processing task (e.g., execution of the RoomNet technology) or application in a commercially reasonable amount of time.

[0232] Code modules or any type of data may be stored on any type of non-transitory computer-readable medium, such as a physical computer storage device including a hard drive, solid-state memory, random access memory (RAM), read-only memory (ROM), optical disk, volatile or non-volatile storage device, a combination of the same or similar thereof, and the like. Methods and modules (or data) may also be sent as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless-based and wired / cable-based media), and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). The results of the disclosed processes or process steps may be stored persistently or otherwise in any type of non-transitory tangible computer storage device, or may be transmitted via a computer-readable transmission medium.

[0233] Any process, frame, state, step or function in the flowchart described herein and / or depicted in the accompanying drawings should be understood to represent a code module, code segment or code portion, which includes one or more executable instructions for implementing a specific function (e.g., logic or arithmetic) or a step in a process. Various processes, blocks, states, steps or functions can be combined, rearranged, added, deleted, modified or otherwise changed with the illustrative examples provided herein. In some embodiments, additional or different computing systems or code modules can perform some or all of the functions described herein. The methods and processes described herein are also not limited to any particular sequence, and the blocks, steps or states associated therewith can be executed in other appropriate sequences, for example, in serial, parallel or in some other manner. Tasks or events can be added to or deleted from the disclosed example embodiments. In addition, the separation of various system components in the implementation described herein is for illustrative purposes and should not be understood as requiring such separation in all implementations. It should be understood that the described program components, methods and systems can generally be integrated together in a single computer product or packaged into multiple computer products. Many implementation variations are possible.

[0234] The processes, methods and systems can be implemented in a network (or distributed) computing environment. Network environments include enterprise-wide computer networks, intranets, local area networks (LANs), wide area networks (WANs), personal area networks (PANs), cloud computing networks, crowd-sourced computing networks, the Internet, and the World Wide Web. The network can be a wired or wireless network or any other type of communication network.

[0235] The systems and methods of the present disclosure each have several innovative aspects, none of which is solely responsible for or requires the desired attributes disclosed herein. The various features and processes described herein can be used independently of each other, or can be combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. It will be apparent to those skilled in the art that various modifications to the implementations described in the present disclosure will be apparent, and the general principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Therefore, the claims are not intended to be limited to the implementations shown herein, but are consistent with the widest scope of the present disclosure, principles, and novel features disclosed herein.

[0236] Certain features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations, either individually or in any suitable subcombination. Furthermore, although the features above may be described as functioning in certain combinations and even initially claimed as such, one or more features from the claimed combination may be excluded from the combination in some cases, and the claimed combination may be directed to a subcombination or variations of the subcombination. Individual features or groups of features are not necessary or essential for each and all embodiments.

[0237] Unless otherwise specified, or understood in the context of use in other ways, conditional language used herein, such as "can", "may", "possibly", "may", "for example", etc., is generally intended to convey that certain embodiments include certain features, elements and / or steps, while other embodiments do not include certain features, elements and / or steps. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements and / or steps in any way, or that one or more embodiments must include logic for determining whether these features, elements and / or steps are included or to be performed in any particular embodiment (regardless of whether there is author input or prompt). The terms "comprise", "include", "have", etc. are synonymous and are used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. In addition, the term "or" is used in its inclusive sense (rather than in its exclusive sense), so when used, for example, to connect a list of elements, the term "or" means one, some or all of the elements in the list. In addition, unless otherwise specified, the articles "a", "an" and "the" used in this application and the appended claims should be understood to mean "one or more" or "at least one".

[0238] As used herein, phrases referring to "at least one" of a list of items refer to any combination of those items, including single members. For example, "at least one of A, B, or C" is intended to cover: A, B, C, A and B, A and C, B and C, and A, B and C. Unless otherwise specifically stated, conjunctive language such as the phrase "at least one of X, Y, and Z" is otherwise understood by context as being generally used to convey that an item, term, etc. may be at least one of X, Y, or Z. Thus, such conjunctive language is generally not intended to imply that certain embodiments require the presence of at least one of X, at least one of Y, and at least one of Z, respectively.

[0239] Similarly, although operations can be depicted in the drawings in a particular order, it should be appreciated that it is not necessary to perform these operations in the particular order shown or in sequence, or to perform all the operations shown, to achieve the desired result. In addition, the drawings may schematically depict one or more example processes in the form of a flow chart. However, other operations not shown may be included in the schematically illustrated example methods and processes. For example, one or more additional operations may be performed before, after, simultaneously, or between any of the illustrated operations. In addition, operations may be rearranged or reordered in other embodiments. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. In addition, other implementations are within the scope of the following claims. In some cases, the actions recorded in the claims may be performed in different orders and still achieve the desired results.

Claims

1. A system comprising: Non-transitory memory configured to store: images of rooms; as well as Neural networks, including: an encoder-decoder subnetwork comprising an encoder and a decoder; and a classifier subnetwork in communication with the encoder-decoder subnetwork and configured to classify a room type associated with the image; a hardware processor in communication with the non-transitory memory, the hardware processor being programmed to: accessing the image; predicting a plurality of two-dimensional keypoint maps corresponding to a plurality of room types using the encoder, the decoder, and the image; determining a predicted room type from the plurality of room types using the encoder, the classifier subnetwork, and the image; and A predicted layout of the room in the image is determined based at least in part on the two-dimensional keypoint map and the predicted room type.

2. The system according to claim 1, wherein: Each of the room types includes a semantic segmentation for a region in the room type, the semantic segmentation including an identification as a floor, a ceiling, or a wall.

3. The system according to claim 1, wherein: A first keypoint sequence is associated with a first room type of the room types, and a second keypoint sequence is associated with a second room type of the room types, wherein the first keypoint sequence and the second keypoint sequence are different.

4. The system according to claim 1, wherein: The encoder includes a plurality of convolutional layers and a plurality of pooling layers, or the decoder includes a plurality of convolutional layers and a plurality of upsampling layers.

5. The system according to claim 1, wherein: The encoder-decoder subnetwork includes multiple recursive layers.

6. The system according to claim 5, wherein: The number of recursive iterations of the plurality of recursive layers is at least two.

7. The system according to claim 1, wherein: The predicted two-dimensional keypoint map includes a heat map.

8. The system according to claim 7, wherein: The hardware processor is programmed to extract keypoint locations from the heat map as maximum values ​​of the heat map.

9. The system according to claim 1, wherein: The hardware processor is programmed to: accessing object information from an object identifier that analyzes the image; and The object information is combined with the predicted layout of the room.

10. The system according to claim 9, wherein: The object identifier is configured to detect a cuboid in the image.

11. A system comprising: a non-transitory memory configured to store parameters for the neural network; as well as a hardware processor in communication with the non-transitory memory, the hardware processor being programmed to: An image is received, wherein the image is associated with: Reference room types from multiple room types, and reference key points associated with the reference room layout; A neural network is generated, the neural network comprising: an encoder-decoder subnetwork configured to output two-dimensional keypoints associated with the predicted room layout, wherein the encoder-decoder subnetwork includes an encoder and a decoder, and a side sub-network in communication with the encoder-decoder network, configured to classify the reference room type associated with the image by outputting a predicted room type from the plurality of room types; and Parameters of the neural network are updated based at least in part on the optimized loss function.

12. The system according to claim 11, wherein: A first keypoint sequence is associated with a first room type of the plurality of room types, and a second keypoint sequence is associated with a second room type of the plurality of room types, wherein the first keypoint sequence is different from the second keypoint sequence.

13. The system according to claim 11, wherein: The encoder and the decoder include a plurality of recursive layers.

14. The system according to claim 11, wherein: The optimization loss function is based at least in part on a first loss for the two-dimensional keypoints and a second loss for the predicted room type.

15. The system of claim 14, wherein: The two-dimensional keypoints are extracted from the predicted heatmap.

16. The system of claim 15, wherein: The hardware processor is programmed to: calculating a reference heat map associated with the reference keypoints; and The first loss for the two-dimensional keypoint is calculated based on a difference between the predicted heat map and the reference heat map.

17. A wearable display system comprising: an outward-facing imaging system configured to acquire images of an environment of a wearer of the wearable display system; a non-transitory memory configured to store the image; as well as a hardware processor in communication with the non-transitory memory, the processor being programmed to: accessing the image; and analyzing the image to determine a predicted layout of a room in the image, wherein, to analyze the image, the processor is programmed to: Determining two-dimensional keypoints associated with a room type associated with the image using a neural network, wherein the neural network comprises: an encoder-decoder subnetwork configured to output the two-dimensional keypoints, and a side sub-network in communication with the encoder-decoder sub-network, configured to classify the room type associated with the image by outputting a room type; and The predicted layout is provided based at least in part on the two-dimensional keypoints and the room type.

18. The wearable display system according to claim 17, wherein: The neural network includes a convolutional encoder-decoder network.

19. The wearable display system according to claim 17, wherein: The neural network includes a classifier configured to determine the room type.

20. A method comprising: Access room images; predicting a two-dimensional keypoint map corresponding to a plurality of room types based at least in part on the image; determining a predicted room type from a plurality of room types based at least in part on the image and a side sub-network configured to classify a room type associated with the image; A predicted layout of the room in the image is determined based at least in part on the two-dimensional keypoint map and the predicted room type.

21. The method according to claim 20, wherein: A two-dimensional keypoint map is also predicted based at least in part on a neural network.

22. The method according to claim 20, wherein: The two-dimensional keypoint map is predicted based at least in part on one or more heat maps.

23. The method according to claim 20, wherein: Each of the room types includes a semantic segmentation for a region in the room type, the semantic segmentation including an identification as a floor, a ceiling, or a wall.

24. The method according to claim 20, wherein: A first keypoint sequence is associated with a first room type of the room types, and a second keypoint sequence is associated with a second room type of the room types, wherein the first keypoint sequence and the second keypoint sequence are different.

Images