Room scan-based plan generation

Abstract

The invention relates to plan generation based on room scans. Various implementations disclosed herein include devices, systems, and methods for generating plan graphs and measurements using a three-dimensional (3D) representation of a physical environment generated based on sensor data.

Description

[0001] This application is a divisional application of the invention patent application with application date of January 15, 2021, application number 202110057602.0, and invention name “Floor plan generation based on room scanning”. Technical Field

[0002] The present disclosure relates generally to generating two-dimensional and three-dimensional geometric representations of a physical environment, and more particularly, to systems, methods, and devices for generating geometric representations based on information detected in the physical environment. Background Art

[0003] Floor plans play an important role in designing, understanding, and reshaping interior spaces. Floor plans often effectively convey geometric and semantic information about the physical environment. For example, users can view floor plans to quickly identify room extents, wall structures and corners, locations of doors and windows, and object placement.

[0004] There are many obstacles to providing a computer-based system to automatically generate floor plans, room measurements, or object measurements based on sensor data. Sensor data (e.g., image and depth data) acquired about the physical environment may be incomplete or insufficient to provide accurate floor plans and measurements. For example, indoor environments often contain various objects, such as lamps, tables, chairs, etc., which may hide the architectural lines of the room, which can be used to detect the edges of the room to construct an accurate floor plan. For another example, image and depth data often lack semantic information, and floor plans and measurements generated without such data may lack accuracy.

[0005] Existing technologies do not allow for automatic, accurate, and efficient generation of floor plans and measurements using a mobile device, e.g., based on capturing photos or videos or other sensor data as a user walks around a room. Furthermore, existing technologies may not be able to provide sufficiently accurate and efficient floor plans and measurements in real time (e.g., immediate floor plans / measurements during scanning) of an environment. Summary of the invention

[0006] Various embodiments disclosed herein include devices, systems, and methods for generating floor plans and measurements using a three-dimensional (3D) representation of a physical environment. A 3D representation of a physical environment can be generated based on sensor data such as images and depth sensor data. In some embodiments, a semantically labeled 3D representation of a physical environment is used to facilitate the generation of floor plans and measurements. Some embodiments perform semantic segmentation and labeling of a 3D point cloud of a physical environment. The technology disclosed herein can achieve various advantages by using a semantic 3D representation (such as a semantically labeled 3D point cloud) encoded into a two-dimensional (2D) lateral domain. Using a semantic 3D representation in a 2D lateral domain can facilitate the efficient identification of structures used to generate floor plans or measurements.

[0007] The floor plan can be provided in various formats. In some embodiments, the floor plan includes a 2D top view of the room. The floor plan can graphically depict the boundaries of the room, for example, by graphically depicting walls, barriers, or other limitations of the room range using lines or other graphical features. The floor plan can graphically depict the location and geometry of wall features (such as wall edges, doors, and windows). The floor plan can graphically depict objects in the room, such as sofas, tables, chairs, appliances, etc. The floor plan may include identifiers that identify the boundaries, walls, doors, windows, and objects in the room, such as text labels or reference numbers that identify such elements. The floor plan may include indications of measurements of the boundaries, wall edges, doors, windows, and objects in the room, such as numbers that specify the length of a wall, the diameter of a table, the width of a window, etc.

[0008] According to some specific implementations, a floor plan is created based on a user performing a room scan (e.g., moving a mobile device to capture images and depth data around the user in the room). Some specific implementations provide a preview of a preliminary 2D floor plan during a room scan. For example, when a user walks around a room where sensor data is captured, the user's device may display a preview of the preliminary 2D floor plan being generated. The preview is "real-time", that is, the preview is provided during the continuous capture of a sensor data stream or sensor data group used to generate the preliminary 2D floor plan. In order to enable real-time preview of the preliminary 2D floor plan, the preview can be generated (at least initially) differently from the final scanned floor plan. In one example, the preview is generated without certain post-processing techniques (e.g., fine-tuning, corner correction, etc.) used to generate the final scanned floor plan. In other examples, the real-time preview can use a neural network with lower computational density than that used to generate the final scanned floor plan. The use of 2D semantic data (e.g., for different layers of the room) can also facilitate making the preview determination effective enough for real-time display.

[0009] In some implementations, a floor plan can be generated based on separately identifying wall structures (e.g., wall edges, doors, and windows) and detecting bounding boxes of objects (e.g., furniture, appliances, etc.). Wall structures and objects can be detected separately and thus combined using different techniques and the results to generate a floor plan representing both wall structures and objects.

[0010] In some implementations, the floor plan creation process identifies wall structures (e.g., wall edges) based on a 2D representation that encodes 3D semantic data in multiple layers. For example, the 3D semantic data may be segmented into multiple horizontal layers that identify where the wall edges of a room are located.

[0011] According to some implementations, different techniques may be used to collect measurements of wall attributes (e.g., walls, doors, and windows) and objects (e.g., furniture, appliances, etc.) of a room. For example, for wall attributes, such as doors and windows, a light intensity image (e.g., an RGB image) may be used to generate boundaries (2D polygonal shapes) in addition to or in lieu of depth data. This may provide various advantages, for example, in situations where depth data may be skewed due to the transparency of windows and doors that may include windows. After determining the 2D polygonal shape from the light intensity image, the depth data or a 3D representation (e.g., a 3D semantic point cloud) based on the depth data may then be used to determine specific measurements of the door or window. In some implementations, an object is measured by first generating a 3D bounding box of the object based on the depth data, refining the bounding box using various neural networks and refinement algorithms described herein, and collecting measurements based on the refined bounding box and associated 3D data points for the corresponding bounding box.

[0012] Some specific implementations of the present disclosure relate to an exemplary method of generating and displaying a real-time preview of a preliminary 2D floor plan. The exemplary method first involves displaying a real-time camera image feed (e.g., real-time video) including a sequence of images of a physical environment at an electronic device having a processor (e.g., a smart phone). For example, when a user captures video while walking around a room to capture images of different parts of the room from multiple perspectives, these images are displayed in real time on the mobile device so that the user sees what he / she is capturing.

[0013] The exemplary method also involves obtaining a 3D representation of the physical environment generated based on depth data and light intensity data acquired during display of a real-time camera feed. For example, a 3D point cloud may be generated based on depth camera information received simultaneously with the image.

[0014] The exemplary method also involves generating a real-time preview of a preliminary 2D floor plan of a physical environment based on a 3D representation of the physical environment. For example, semantic information may be included in or associated with a 3D point cloud, and 2D semantic data (e.g., in layers) may be semantically generated from the 3D point cloud. Additionally, the 2D semantic data may be used to identify walls and wall attributes or features (e.g., doors and windows) for real-time preview. Furthermore, a representation of an object in the real-time preview may be generated based on a 3D bounding box determined using the 3D point cloud.

[0015] The exemplary method also involves displaying a live preview of the preliminary 2D floor plan simultaneously with the live camera feed. For example, while a user is viewing a live camera feed of a room environment, another viewing window with the 2D floor plan may be overlaid on top of the live camera feed (e.g., picture-in-picture (PIP)) when generated.

[0016] In some implementations, the exemplary method further involves generating a final 2D floor plan of the physical environment based on the 3D representation, wherein generating the final 2D floor plan uses a different process from generating the real-time preview of the preliminary 2D floor plan. For example, the different process may use a more computationally intensive neural network with fine-tuning (e.g., corner correction), etc. In some implementations, the different process includes segmenting corners and small walls based on the 3D representation using a more computationally intensive neural network, generating a transitional 2D floor plan based on the segmented corners and small walls, determining a refinement scheme for the transitional 2D floor plan using a standardized algorithm, and generating the final 2D floor plan of the physical environment based on the determined refinement scheme for the transitional 2D floor plan.

[0017] In some implementations, the exemplary method further involves generating an edge map by identifying walls in the physical environment based on the 3D representation, updating the edge map by identifying wall attributes (e.g., doors and windows) in the physical environment based on the 3D representation, updating the edge map by identifying objects in the physical environment based on the 3D representation, and generating a real-time preview of the preliminary 2D floor plan based on the updated edge map including the identified walls, the identified wall attributes, and the identified objects. In some implementations, generating a real-time preview of the 2D floor plan includes generating 2D semantic data for multiple horizontal layers of the physical environment based on the 3D representation, and generating the 2D floor plan using the 2D semantic data. For example, each layer provides x, y semantics for a range of z values, for example, for each x, y position in the z value range of 0-10, the first layer can be the most common semantic label.

[0018] In some implementations, generating an edge map by identifying walls also includes determining a parametric refinement line of the edge map using a line fitting algorithm, and updating the edge map based on the parametric refinement line. In some implementations, updating the edge map by identifying wall attributes includes determining the boundaries of the identified wall attributes using a wall attribute neural network and an image sequence fed by a real-time camera (e.g., RGB data of a transparent window), and generating a refined boundary based on a 3D representation associated with the identified wall attributes using a polygon heuristic algorithm. In some implementations, updating the edge map by identifying objects includes generating a 3D bounding box corresponding to the identified object in the physical environment based on the 3D representation, and generating a 2D representation of the 3D bounding box (e.g., a furniture icon or a flat 2D bounding box).

[0019] In some implementations, the 3D representation is associated with 3D semantic data including a 3D point cloud, the 3D point cloud including semantic tags associated with at least a portion of the 3D points within the 3D point cloud. Additionally, in some implementations, the semantic tags identify walls, wall attributes (e.g., doors and windows), objects, and partitions of objects of the physical environment.

[0020] Some specific implementations of the present disclosure involve an exemplary method for generating and displaying a 2D floor plan. The exemplary method first involves obtaining a 3D representation of a physical environment, which is generated based on depth data and light intensity image data acquired during a scanning process. For example, a 3D point cloud can be generated based on depth camera information received simultaneously with the image during a room scan. For example, an algorithm can be used for semantic segmentation and labeling of a 3D point cloud of an indoor scene, where objects in the point cloud can have significant variations and complex configurations.

[0021] The exemplary method also involves detecting the location of a wall structure in a physical environment based on a 3D representation. For example, a wall can be identified by generating 2D semantic data (e.g., in layers), using the 2D semantic data to generate an edge map using a neural network, and determining vector parameters to normalize the edge map in a 3D normalized plan view. Wall attributes or wall properties (e.g., doors / windows) can be identified based on an RGB image and depth data to generate a polygonal boundary. This technique for doors and windows provides advantages, particularly since the transparency of windows may produce noise / errors in the depth data.

[0022] The exemplary method also involves generating bounding boxes corresponding to objects in the physical environment based on the 3D representation. For example, the 3D bounding box can provide the location, position (e.g., location and orientation), and shape of each piece of furniture and appliance in the room. The bounding box can be refined using RGB data and a novel multi-network training technique (e.g., 2-stage neural network fine-tuning for low precision / high recall and high precision / low recall).

[0023] The exemplary method also involves displaying a 2D plan view that provides a view (eg, a top view) of the physical environment. In some implementations, the 2D plan view is determined based on the location of the wall structure and the bounding box corresponding to the object.

[0024] In some embodiments, detecting the location of a wall structure in a physical environment based on a 3D representation includes identifying walls and wall attributes (e.g., doors and windows) of the physical environment from the wall structure based on the 3D representation, and generating an edge map of the identified walls and wall attributes based on the 3D representation, wherein the 2D plan view is based on the generated edge map including the identified walls and the identified wall attributes. In some embodiments, the exemplary method further involves segmenting corners and small walls using a more computationally dense neural network based on the 3D representation, generating a transitional 2D plan view based on the segmented corners and small walls, determining a refinement scheme for the transitional 2D plan view using a normalization algorithm, and generating a final 2D plan view of the physical environment based on the determined refinement scheme for the transitional 2D plan view. In some embodiments, the exemplary method further involves determining the boundary of the identified wall structure using a wall structure neural network and light intensity image data (e.g., RGB data) acquired during a scanning process, and generating a refined boundary based on 3D semantic data associated with the identified wall attributes using a polygon heuristic algorithm.

[0025] In some implementations, the bounding boxes are refined bounding boxes, and the exemplary method further involves generating a refined bounding box for an object by generating a proposed bounding box using a first neural network, and generating a refined bounding box by identifying features of the object using a second neural network (e.g., low precision / high recall to generate features of the object) and refining the proposed bounding box based on the identified features using a third neural network (e.g., high precision / low recall to refine the accuracy of the generated features and output a refined bounding box). In some implementations, the first neural network generates the proposed bounding box based on a 3D representation associated with the object. In some implementations, the second neural network identifies features of the object based on a 3D representation associated with the object and light intensity image data (e.g., RGB data) acquired during a scanning process. In some implementations, the third neural network is trained to refine the accuracy of the identified features from the second neural network, and outputs a refined bounding box based on the 3D representation associated with the object and light intensity image data (e.g., RGB data) acquired during a scanning process. In some implementations, a bounding box provides location information, position information (eg, location and orientation information), and shape information of an object in a physical environment.

[0026] In some implementations, the 3D representation is associated with 3D semantic data including a 3D point cloud, the 3D point cloud including semantic tags associated with at least a portion of the 3D points within the 3D point cloud. Additionally, in some implementations, the semantic tags identify walls, wall attributes (e.g., doors and windows), objects, and partitions of objects of the physical environment.

[0027] Some specific implementations of the present disclosure relate to an exemplary method for providing a floor plan based on 2D semantic data. The exemplary method first involves acquiring 3D semantic data of a physical environment, which is generated based on depth data and light intensity image data acquired during a scanning process. For example, a 3D point cloud can be generated based on depth camera information received simultaneously with the image during a room scan. For example, an algorithm can be used for semantic segmentation and labeling of a 3D point cloud of an indoor scene, where objects in the point cloud can have significant variations and complex configurations.

[0028] The exemplary method also involves generating multiple horizontal layers of 2D semantic data of the physical environment based on the 3D semantic data. For example, each layer provides x, y semantics for a range of z values, for example, for each x, y position in the z value range of 0-10, the first layer can be the most common semantic label.

[0029] The exemplary method also involves providing a floor plan based on generating an edge map using the 2D semantic data, wherein the floor plan provides a view (e.g., a top view) of the physical environment. In some implementations, generating the edge map may involve determining a parameter representation and / or vector parameters to normalize the edge map in a 3D normalized floor plan.

[0030] In some embodiments, providing a floor plan further comprises generating an edge map by identifying walls in a physical environment based on 2D semantic data of multiple horizontal layers, updating the edge map by identifying wall attributes (e.g., doors and windows) in the physical environment based on 3D semantic data, updating the edge map by identifying objects in the physical environment based on the 3D semantic data, and generating a floor plan based on the updated edge map including the identified walls, the identified wall attributes, and the identified objects.

[0031] In some implementations, the identified wall is a floor-to-ceiling wall (e.g., not a partition wall), wherein identifying the floor-to-ceiling wall based on the 2D semantic data of the multiple horizontal layers includes identifying a floor of the physical environment having a lowest level of the multiple horizontal layers, identifying a ceiling of the physical environment having a highest level of the multiple horizontal layers, determining that the particular identified wall is not a floor-to-ceiling wall (e.g., a partition wall) based on the particular identified wall not satisfying a height threshold compared to the height of the ceiling, and updating the edge map by removing the particular identified wall from the edge map. In some implementations, generating an edge map by identifying the wall also includes determining a parametric refinement line of the edge map using a line fitting algorithm, and updating the edge map based on the parametric refinement line. In some implementations, updating the edge map by identifying wall attributes includes determining a boundary of the identified wall attribute using a wall attribute neural network and a light intensity image (e.g., RGB data of a transparent window) acquired during a scanning process, and generating the refined boundary based on 3D semantic data associated with the identified wall attribute using a polygon heuristic algorithm.

[0032] In some implementations, updating the edge map by recognizing the object includes generating a 3D bounding box corresponding to the recognized object in the physical environment based on the 3D semantic data, and generating a 2D representation of the 3D bounding box (e.g., a furniture icon or a flat 2D bounding box). In some implementations, the bounding boxes are refined bounding boxes, and generating the refined bounding boxes of the objects includes generating proposed bounding boxes using a first neural network, and generating the refined bounding boxes by recognizing features of the object using a second neural network (e.g., low precision / high recall to generate features of the object) and refining the proposed bounding boxes based on the recognized features using a third neural network (e.g., high precision / low recall to refine the accuracy of the generated features and output a refined bounding box).

[0033] In some implementations, the 3D semantic data includes semantic tags associated with at least a portion of the 3D points within the 3D point cloud representation of the physical environment. In some implementations, the semantic tags identify walls, wall attributes (e.g., doors and windows), objects, and partitions of objects of the physical environment.

[0034] Some implementations of the present disclosure relate to an exemplary method for providing measurement data of objects and wall structures within a physical environment. The exemplary method first involves obtaining a 3D representation of the physical environment generated based on depth data acquired during a scanning process. For example, a 3D point cloud can be generated based on depth camera information received simultaneously with an image. In some implementations, the 3D representation is associated with 3D semantic data. For example, an algorithm can be used for semantic segmentation and labeling of a 3D point cloud of an indoor scene, where objects in the point cloud can have significant variations and complex configurations.

[0035] The exemplary method also involves generating 2D boundaries of wall attributes (eg, doors and windows) in the physical environment based on a light intensity image (eg, an RGB image) of the physical environment.

[0036] The exemplary method also involves providing measurements of wall attributes based on the 2D boundary and the 3D representation. For example, the 3D representation is used to determine the depth and / or width of a wall attribute such as a door or window given to a 2D polygon shape associated with the wall attribute.

[0037] The exemplary method also involves generating 3D bounding boxes corresponding to objects in the physical environment based on the 3D representation. For example, the 3D bounding box can provide the location, position (e.g., location and orientation), and shape of each piece of furniture and appliance in the room. The bounding box can be refined using RGB data and a novel multi-network conditioning technique.

[0038] The exemplary method also involves providing measurements of a 3D bounding box representing measurements of the corresponding object. For example, the length, width, and height of the bounding box correspond to the length, width, and height of the object.

[0039] In some implementations, the bounding boxes are refined bounding boxes, and the exemplary method further involves generating a refined bounding box for an object by generating a proposed bounding box using a first neural network, and generating a refined bounding box by identifying features of the object using a second neural network (e.g., low precision / high recall to generate features of the object) and refining the proposed bounding box based on the identified features using a third neural network (e.g., high precision / low recall to refine the accuracy of the generated features and output a refined bounding box). In some implementations, the first neural network generates the proposed bounding box based on a 3D representation associated with the object. In some implementations, the second neural network identifies features of the object based on a 3D representation associated with the object and light intensity image data (e.g., RGB data) acquired during a scanning process. In some implementations, the third neural network is trained to refine the accuracy of the identified features from the second neural network, and outputs a refined bounding box based on the 3D representation associated with the object and light intensity image data (e.g., RGB data) acquired during a scanning process. In some implementations, a bounding box provides location information, position information (eg, location and orientation information), and shape information of an object in a physical environment.

[0040] In some implementations, the exemplary method further involves generating a refined boundary of the wall attribute based on the 3D semantic data associated with the wall attribute using a polygon heuristic algorithm. In some implementations, the wall attribute includes a door or a window.

[0041] In some implementations, the measurements of the bounding box associated with the measurement of a particular wall attribute include the length, width, and height of the particular wall attribute. For example, the length, width, and height of a door. In some implementations, the measurements of the 3D bounding box for a particular object include the length, width, and height corresponding to the length, width, and height of the particular object. For example, the length, width, and height of a bounding box generated for a table or chair in a room.

[0042] In some implementations, the 3D representation includes a 3D point cloud, and the associated 3D semantic data includes semantic tags associated with at least a portion of the 3D points within the 3D point cloud. In some implementations, the semantic tags identify walls, wall attributes (e.g., doors and windows), objects, and partitions of the object of the physical environment.

[0043] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0045] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0046] Figure 2 is a block diagram of an exemplary server according to some specific implementations.

[0047] Figure 3 is a block diagram of an exemplary device according to some implementations.

[0048] Figure 4 is a system flow diagram for exemplary generation of a semantic three-dimensional (3D) representation using 3D data and semantic segmentation based on depth and light intensity image information according to some specific implementations.

[0049] Figure 5is a flowchart representation of an exemplary method for generating and displaying a real-time preview of a two-dimensional (2D) floor plan of a physical environment based on a 3D representation of the physical environment, according to some implementations.

[0050] Figure 6 is a system flow diagram for exemplary generation of a real-time preview of a 2D floor plan of a physical environment based on a 3D representation of the physical environment, according to some implementations.

[0051] Figure 7 is a flowchart representation of an exemplary method for generating and displaying a 2D floor plan of a physical environment according to some implementations.

[0052] FIG. 8A to FIG. 8D is a system flow diagram illustrating exemplary generation of a 2D floor plan of a physical environment according to some implementations.

[0053] Fig. 9 is a flowchart representation of an exemplary method for generating and providing a floor plan of a physical environment based on generating an edge map using 2D semantic data according to some specific implementations.

[0054] Fig.10 is a system flow diagram for exemplary generation of a floor plan of a physical environment based on generating an edge map using 2D semantic data according to some specific implementations.

[0055] Fig.11 is a flowchart representation of an exemplary method for generating and providing measurements of a wall structure based on measurements of a 2D boundary and a 3D representation and a 3D bounding box that represents the measurements of the corresponding object, according to some implementations.

[0056] Fig. 12A is a system flow diagram of an exemplary generation of measurements of a wall structure based on a 2D boundary and a 3D representation according to some specific implementations.

[0057] Fig. 12B is a system flow diagram for exemplary generation of measurements of a 3D bounding box representing measurements of a corresponding object according to some specific implementations.

[0058] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. DETAILED DESCRIPTION

[0059] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0060] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. In this example, the exemplary operating environment 100 shows an exemplary physical environment 105 including walls 130, 132, 134, a chair 140, a table 142, a door 150, and a window 152. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a server 110 and a device 120. In the exemplary implementation, the operating environment 100 does not include the server 110, and the methods described herein are performed on the device 120.

[0061] In some implementations, the server 110 is configured to manage and coordinate the user experience. In some implementations, the server 110 includes a suitable combination of software, firmware, and / or hardware. Figure 2 The server 110 is described in more detail. In some implementations, the server 110 is a computing device located locally or remotely relative to the physical environment 105. In one example, the server 110 is a local server located within the physical environment 105. In another example, the server 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the physical environment 105. In some implementations, the server 110 is communicatively coupled to the device 120 via one or more wired or wireless communication channels (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.).

[0062] In some implementations, the device 120 is configured to present an environment to a user. In some implementations, the device 120 includes a suitable combination of software, firmware, and / or hardware. Figure 3 Device 120 is described in more detail. In some implementations, the functionality of server 110 is provided by and / or combined with device 120.

[0063] In some implementations, device 120 is a handheld electronic device (e.g., a smartphone or tablet) configured to present content to a user. In some implementations, the user wears device 120 on his / her head. Device 120 may include one or more displays provided for displaying content. For example, device 120 may surround the user's field of view. In some implementations, device 120 is replaced with a chamber, housing, or chamber configured to present content, where the user does not wear or hold the device 120.

[0064] Figure 2 is a block diagram of an example of a server 110 according to some implementations. Although some specific features are shown, those skilled in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some specific implementations, the server 110 includes one or more processing units 202 (e.g., a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a graphics processing unit (GPU), a central processing unit (CPU), a processing core, etc.), one or more input / output (I / O) devices 206, one or more communication interfaces 208 (e.g., a universal serial bus (USB), FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Global Positioning System (GPS), infrared (IR), Bluetooth, ZIGBEE and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 210, a memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

[0065] In some implementations, the one or more communication buses 204 include circuits for interconnecting system components and controlling communications between system components. In some implementations, the one or more I / O devices 206 include at least one of a keyboard, a mouse, a touch pad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

[0066] The memory 220 includes a high-speed random access memory, such as a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate random access memory (DDR RAM), or other random access solid-state memory devices. In some specific implementations, the memory 220 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. The memory 220 optionally includes one or more storage devices located away from the one or more processing units 202. The memory 220 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 220 or the non-transitory computer-readable storage medium of the memory 220 stores the following programs, modules, and data structures, or a subset thereof, including an optional operating system 230 and one or more application programs 240.

[0067] The operating system 230 includes processes for handling various basic system services and for performing hardware-related tasks. In some implementations, the application 240 is configured to manage and coordinate one or more experiences of one or more users (e.g., a single experience of one or more users, or multiple experiences of corresponding groups of one or more users).

[0068] Application 240 includes 3D representation unit 242, real-time preview unit 244, floor plan unit 246, and measurement unit 248. 3D representation unit 242, real-time preview unit 244, floor plan unit 246, and measurement unit 248 may be combined into a single application or unit, or separated into one or more additional applications or units.

[0069] The 3D representation unit 242 is configured with instructions that can be executed by the processor to acquire image data (e.g., light intensity data, depth data, etc.) and integrate (e.g., fuse) the image data using one or more of the techniques disclosed herein. For example, the 3D representation unit 242 fuses an RGB image from a light intensity camera with a sparse depth map from a depth camera (e.g., a time-of-flight sensor) and other physical environment information sources to output dense depth point cloud information. In addition, the 3D representation unit 242 is configured with instructions that can be executed by the processor to acquire light intensity image data (e.g., RGB) and execute a semantic segmentation algorithm to assign semantic tags to identification features in the image data using one or more of the techniques disclosed herein and generate semantic image data (e.g., RGB-S). The 3D representation unit 242 is further configured with instructions that can be executed by the processor to acquire light intensity image data (e.g., RGB) and depth image data, and use one or more of the techniques disclosed herein to generate a semantic 3D representation (e.g., a 3D point cloud with associated semantic tags). In some implementations, the 3D representation unit 242 includes separate units, such as an integration unit for generating 3D point cloud data, a semantic unit for performing semantic segmentation based on light intensity data (e.g., RGB-S), and a semantic 3D unit for generating a semantic 3D representation, as described herein with reference to Figure 4 Further described.

[0070] The real-time preview unit 244 is configured with instructions that can be executed by the processor to generate and display a real-time preview of a 2D floor plan of the physical environment based on a 3D representation of the physical environment (e.g., 3D point cloud, 3D mesh reconstruction, semantic 3D point cloud, etc.) using one or more of the techniques disclosed herein. The 2D floor plan preview is then overlaid onto the real-time camera feed for picture-in-picture display on the device. For example, the real-time preview unit 244 acquires a sequence of light intensity images (e.g., a real-time camera feed), a semantic 3D representation (e.g., a semantic 3D point cloud) generated from the 3D representation unit 242, and other sources of physical environment information (e.g., camera positioning information from a camera simultaneous localization and mapping (SLAM) system) from a light intensity camera to output a 2D floor plan image that is iteratively updated using the sequence of light intensity images. In order to generate the 2D floor plan, the real-time preview unit 244 is configured with instructions that can be executed by the processor to generate an edge map of the wall identified in the sequence of light intensity images based on the semantic 3D representation and perform post-processing using a line fitting algorithm. The real-time preview unit 244 is further configured with instructions executable by the processor to identify wall attributes (eg, doors and windows) in the sequence of light intensity images based on the semantic 3D representation and perform post-processing using the fine-tuning algorithm techniques further disclosed herein.

[0071] The real-time preview unit 244 may also be configured with instructions that are executable by the processor to identify objects (e.g., furniture, appliances, etc.) in the light intensity image sequence based on a semantic 3D representation, generate a bounding box for each identified object, and perform post-processing using the fine-tuning algorithm techniques further disclosed herein.

[0072] The real-time preview unit 244 generates a 2D plan view from the edge map, the boundaries of the identified wall attributes, and the bounding boxes of the identified objects using one or more processes disclosed further herein.

[0073] In some implementations, the real-time preview unit 244 includes independent units, such as an edge mapping unit and associated post-processing units for identifying walls and generating and fine-tuning edge maps, a wall attribute unit and associated post-processing units for identifying and fine-tuning the boundaries of each identified wall attribute, an object detection unit and associated post-processing units for identifying and fine-tuning the bounding box of each identified object, and as described herein with reference to Figure 6 Further described is a floor plan preview unit for generating a 2D floor plan.

[0074] The floor plan unit 246 is configured with instructions that can be executed by the processor to generate and display a 2D floor plan of the physical environment based on a 3D representation of the physical environment (e.g., 3D point cloud, 3D mesh reconstruction, semantic 3D point cloud, etc.) using one or more of the techniques disclosed herein. For example, the floor plan unit 246 acquires a sequence of light intensity images from a light intensity camera (e.g., a real-time camera feed), a semantic 3D representation generated from the 3D representation unit 242 (e.g., a semantic 3D point cloud), and other sources of physical environment information (e.g., camera positioning information from the camera's SLAM system) to output a finalized 2D floor plan image (e.g., a standardized and normalized floor plan). In addition, the floor plan unit 246 generates an edge map of the wall identified in the sequence of light intensity images based on the semantic 3D representation, and performs post-processing using a line fitting algorithm technique and corner fine-tuning using a small wall neural network further disclosed herein. The real-time preview unit 244 is further configured with instructions executable by the processor to identify wall attributes (e.g., doors and windows) in the sequence of light intensity images and perform post-processing using a fine-tuning algorithm technique based on the semantic 3D representation further disclosed herein. The floor plan unit 246 is also configured with instructions executable by the processor to identify objects (e.g., furniture, appliances, etc.) in the sequence of light intensity images based on the semantic 3D representation, generate a bounding box for each identified object, and perform post-processing using a 2-stage fine-tuning neural network technique further disclosed herein. The floor plan unit 246 is further configured with instructions executable by the processor to generate a finalized 2D floor plan from the edge map, the boundaries of the identified wall attributes, and the bounding boxes of the identified objects using one or more processes further disclosed herein.

[0075] In some embodiments, the floor plan unit 246 is further configured with instructions executable by the processor to generate measurement data based on 3D representations of walls identified on the edge map, measurement data of boundaries of identified wall attributes, and measurement data of bounding boxes of identified objects using one or more processes further disclosed herein.

[0076] In some embodiments, the floor plan unit 246 includes independent units, such as an edge mapping unit and associated post-processing units for identifying walls and generating and fine-tuning an edge map with small walls and corners, a wall attribute unit and associated post-processing units for identifying and fine-tuning the boundaries of each identified wall attribute, an object detection unit and associated post-processing units for identifying and fine-tuning the bounding box of each identified object, a floor plan finalization unit for generating a standardized 2D floor plan, and a measurement unit for generating measurement data, as further described herein with reference to Figures 8 and 12.

[0077] The measurement unit 248 is configured with instructions that can be executed by the processor to generate measurement data based on a 3D representation of the wall identified on the edge map (e.g., 3D point cloud, 3D mesh reconstruction, semantic 3D point cloud, etc.), measurement data of the boundaries of the identified wall attributes, and measurement data of the bounding box of the identified object using one or more techniques disclosed herein. For example, the measurement unit 248 obtains the finalized edge map of the wall and the associated depth data, the 2D contours of the identified wall attributes and the associated depth data, and the bounding box of the identified object (e.g., a refined bounding box) from the floor plan unit 244. The measurement unit 248 is configured with instructions that can be executed by the processor to generate measurement data based on the 3D representation of the wall identified on the edge map, measurement data of the boundaries of the identified wall attributes, and measurement data of the bounding box of the identified object using one or more processes further disclosed herein with reference to Figures 8 and 12.

[0078] Although these elements are shown as residing on a single device (e.g., server 110), it should be understood that in other implementations, any combination of elements may be located in separate computing devices. Figure 2 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 2 Some functional modules shown separately in the figure may be implemented in a single module, and the various functions of a single functional block may be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0079] Figure 3is a block diagram of an example of a device 120 according to some implementations. While some specific features are shown, those skilled in the art will appreciate from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the implementations disclosed herein. To this end, as a non-limiting example, in some embodiments, the device 120 includes one or more processing units 302 (e.g., microprocessors, ASICs, FPGAs, GPUs, CPUs, processing cores, etc.), one or more input / output (I / O) devices and sensors 306, one or more communication interfaces 308 (e.g., USB, FIREWIRE, THUNDERBOLT, IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, GSM, CDMA, TDMA, GPS, IR, BLUETOOTH, ZIGBEE, SPI, I2C, and / or similar types of interfaces), one or more programming (e.g., I / O) interfaces 310, one or more AR / VR displays 312, one or more internal-facing and / or external-facing image sensors 314, a memory 320, and one or more communication buses 304 for interconnecting these and various other components.

[0080] In some implementations, the one or more communication buses 304 include circuits that interconnect and control communications between system components. In some implementations, the one or more I / O devices and sensors 306 include at least one of the following: an inertial measurement unit (IMU), an accelerometer, a magnetometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., a blood pressure monitor, a heart rate monitor, a blood oxygen sensor, a blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, one or more depth sensors (e.g., structured light, time of flight, etc.), etc.

[0081] In some implementations, one or more displays 312 are configured to present an experience to a user. In some implementations, one or more displays 312 correspond to holographic, digital light processing (DLP), liquid crystal display (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistor (OLET), organic light emitting diode (OLED), surface conduction electron emitter display (SED), field emission display (FED), quantum dot light emitting diode (QD-LED), microelectromechanical system (MEMS), and / or similar display types. In some implementations, one or more displays 312 correspond to diffractive, reflective, polarized, holographic, etc. waveguide displays. For example, device 120 includes a single display. As another example, device 120 includes a display for each eye of the user.

[0082] In some implementations, the one or more image sensor systems 314 are configured to acquire image data corresponding to at least a portion of the physical environment 105. For example, the one or more image sensor systems 314 include one or more RGB cameras (e.g., with complementary metal oxide semiconductor (CMOS) image sensors or charge coupled device (CCD) image sensors), monochrome cameras, infrared cameras, event-based cameras, etc. In various implementations, the one or more image sensor systems 314 also include an illumination source that emits light, such as a flash. In various implementations, the one or more image sensor systems 314 also include an on-camera image signal processor (ISP) that is configured to perform multiple processing operations on the image data, the multiple processing operations including at least a portion of the processes and techniques described herein.

[0083] The memory 320 includes a high-speed random access memory, such as DRAM, SRAM, DDR RAM or other random access solid-state memory devices. In some specific implementations, the memory 320 includes a non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 320 optionally includes one or more storage devices located away from the one or more processing units 302. The memory 320 includes a non-transitory computer-readable storage medium. In some specific implementations, the memory 320 or the non-transitory computer-readable storage medium of the memory 320 stores the following programs, modules and data structures, or a subset thereof, including an optional operating system 330 and one or more application programs 340.

[0084] The operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some specific implementations, the application 340 is configured to manage and coordinate one or more experiences of one or more users (e.g., a single experience of one or more users, or multiple experiences of corresponding groups of one or more users). The application 340 includes a 3D representation unit 342, a real-time preview unit 344, a floor plan unit 346, and a measurement unit 348. The 3D representation unit 342, the real-time preview unit 344, the floor plan unit 346, and the measurement unit 348 can be combined into a single application or unit, or divided into one or more additional applications or units.

[0085] The 3D representation unit 342 is configured with instructions that can be executed by a processor to acquire image data (e.g., light intensity data, depth data, etc.) and integrate (e.g., fuse) the image data using one or more of the techniques disclosed herein. For example, the 3D representation unit 342 fuses an RGB image from a light intensity camera with a sparse depth map from a depth camera (e.g., a time-of-flight sensor) and other physical environment information sources to output dense depth point cloud information. In addition, the 3D representation unit 342 is configured with instructions that can be executed by a processor to acquire light intensity image data (e.g., RGB) and execute a semantic segmentation algorithm to assign semantic tags to identification features in the image data using one or more of the techniques disclosed herein and generate semantic image data (e.g., RGB-S). The 3D representation unit 342 is further configured with instructions that can be executed by a processor to acquire light intensity image data (e.g., RGB) and depth image data, and use one or more of the techniques disclosed herein to generate a semantic 3D representation (e.g., a 3D point cloud with associated semantic tags). In some implementations, the 3D representation unit 342 includes separate units, such as an integration unit for generating 3D point cloud data, a semantic unit for performing semantic segmentation based on light intensity data (e.g., RGB-S), and a semantic 3D unit for generating a semantic 3D representation, as described herein with reference to Figure 4 Further described.

[0086] The real-time preview unit 344 is configured with instructions that can be executed by the processor to generate and display a real-time preview of a 2D floor plan of the physical environment based on a 3D representation of the physical environment (e.g., 3D point cloud, 3D mesh reconstruction, semantic 3D point cloud, etc.) using one or more of the techniques disclosed herein. The 2D floor plan is then overlaid onto the real-time camera feed for picture-in-picture display. For example, the real-time preview unit 344 acquires a sequence of light intensity images (e.g., a real-time camera feed), a semantic 3D representation (e.g., a semantic 3D point cloud) generated from the 3D representation unit 342, and other sources of physical environment information (e.g., camera positioning information from a camera simultaneous localization and mapping (SLAM) system) from a light intensity camera to output a 2D floor plan image that is iteratively updated using the sequence of light intensity images. To generate the 2D floor plan, the real-time preview unit 344 is configured with instructions that can be executed by the processor to generate an edge map of the wall identified in the sequence of light intensity images based on the semantic 3D representation and perform post-processing using the line fitting algorithm technique further disclosed herein. The real-time preview unit 344 is further configured with instructions executable by the processor to identify wall attributes (eg, doors and windows) in the sequence of light intensity images based on the semantic 3D representation and perform post-processing using the fine-tuning algorithm techniques further disclosed herein.

[0087] The real-time preview unit 344 may also be configured with instructions that are executable by the processor to identify objects (e.g., furniture, appliances, etc.) in the light intensity image sequence based on the semantic 3D representation, generate a bounding box for each identified object, and perform post-processing using the fine-tuning algorithm techniques further disclosed herein.

[0088] The real-time preview unit 344 generates a 2D plan view from the edge map, the boundaries of the identified wall attributes, and the bounding boxes of the identified objects using one or more processes disclosed further herein.

[0089] In some implementations, the real-time preview unit 344 includes independent units, such as an edge mapping unit and associated post-processing units for identifying walls and generating and fine-tuning edge maps, a wall attribute unit and associated post-processing units for identifying and fine-tuning the boundaries of each identified wall attribute, an object detection unit and associated post-processing units for identifying and fine-tuning the bounding box of each identified object, and as described herein with reference to Figure 6 Further described is a floor plan preview unit for generating a 2D floor plan.

[0090] The floor plan unit 346 is configured with instructions that can be executed by the processor to generate and display a 2D floor plan of the physical environment based on a 3D representation of the physical environment (e.g., 3D point cloud, 3D mesh reconstruction, semantic 3D point cloud, etc.) using one or more of the techniques disclosed herein. For example, the floor plan unit 346 acquires a sequence of light intensity images from a light intensity camera (e.g., a real-time camera feed), a semantic 3D representation generated from the 3D representation unit 342 (e.g., a semantic 3D point cloud), and other sources of physical environment information (e.g., camera positioning information from the camera's SLAM system) to output a finalized 2D floor plan image (e.g., a standardized and normalized floor plan). In addition, the floor plan unit 346 is configured with instructions that can be executed by the processor to generate an edge map of the wall identified in the light intensity image sequence based on the semantic 3D representation, and perform post-processing using a line fitting algorithm technique and corner fine-tuning using a small wall neural network further disclosed herein. The real-time preview unit 344 is also configured with instructions executable by the processor to identify wall attributes (eg, doors and windows) in the sequence of light intensity images and perform post-processing using fine-tuning algorithmic techniques based on the semantic 3D representation disclosed further herein.

[0091] The floor plan unit 346 is also configured with instructions that are executable by the processor to identify objects (e.g., furniture, appliances, etc.) in the light intensity image sequence based on the semantic 3D representation, generate a bounding box for each identified object, and perform post-processing using the 2-stage fine-tuned neural network technique further disclosed herein.

[0092] The floor plan unit 346 is further configured with instructions executable by the processor to generate a finalized 2D floor plan from the edge map, the boundaries of the identified wall attributes, and the bounding boxes of the identified objects using one or more processes further disclosed herein.

[0093] In some embodiments, the floor plan unit 346 is further configured with instructions executable by the processor to generate measurement data based on 3D representations of walls identified on the edge map, measurement data of boundaries of identified wall attributes, and measurement data of bounding boxes of identified objects using one or more processes further disclosed herein.

[0094] In some embodiments, the floor plan unit 346 includes independent units, such as an edge mapping unit and associated post-processing units for identifying walls and generating and fine-tuning an edge map with small walls and corners, a wall attribute unit and associated post-processing units for identifying and fine-tuning the boundaries of each identified wall attribute, an object detection unit and associated post-processing units for identifying and fine-tuning the bounding box of each identified object, a floor plan finalization unit for generating a standardized 2D floor plan, and a measurement unit for generating measurement data, as further described herein with reference to Figures 8 and 12.

[0095] The measurement unit 348 is configured with instructions that can be executed by the processor to generate measurement data based on the 3D representation of the wall identified on the edge map, measurement data of the boundaries of the identified wall attributes, and measurement data of the bounding box of the identified object using one or more techniques disclosed herein. For example, the measurement unit 348 obtains the finalized edge map of the wall and the associated depth data, the 2D outlines of the identified wall attributes and the associated depth data, and the bounding box of the identified object (e.g., the refined bounding box) from the floor plan unit 344. The measurement unit 348 is configured with instructions that can be executed by the processor to generate measurement data based on the 3D representation of the wall identified on the edge map, measurement data of the boundaries of the identified wall attributes, and measurement data of the bounding box of the identified object using one or more processes further disclosed herein with reference to Figures 8 and 12.

[0096] Although these elements are shown as residing on a single device (e.g., device 120), it should be understood that in other implementations, any combination of the elements may be located in separate computing devices. Figure 3 It is more of a functional description of various features present in a particular implementation, as opposed to a schematic diagram of the implementations described herein. As one of ordinary skill in the art will recognize, items shown separately may be combined, and some items may be separated. For example, Figure 3Some functional modules (e.g., application 340) shown separately in the figure can be implemented in a single module, and the various functions of a single functional block can be implemented by one or more functional blocks in various specific implementations. The actual number of modules and the division of specific functions and how the features are distributed among them will vary depending on the specific implementation, and in some specific implementations, it depends in part on the specific combination of hardware, software and / or firmware selected for a particular embodiment.

[0097] Figure 4 is a system flow diagram of an exemplary environment 400 in which the system can generate a semantic 3D representation using 3D data and semantic segmentation data based on depth and light intensity image information detected in a physical environment. In some implementations, the system flow of the exemplary environment 400 is performed on a device (e.g., Figures 1 to 3 The system flow of the exemplary environment 400 may be displayed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., Figure 1 and Figure 3 The system flow of the exemplary environment 400 is executed on a device 120 such as a head mounted display (HMD). In some implementations, the system flow of the exemplary environment 400 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 400 is executed on a processor that executes code stored in a non-transitory computer readable medium (e.g., a memory).

[0098] The system flow of the exemplary environment 400 collects the physical environment (e.g., Figure 1 The image data of the physical environment 105) and the 3D representation unit 410 (e.g., Figure 2 3D representation unit 242 and / or Figure 3The 3D representation unit 342 of the present invention uses the 3D point cloud with associated semantic tags to generate a semantic 3D representation 445 representing a surface in the 3D environment. In some embodiments, the semantic 3D representation 445 is a 3D reconstruction mesh using a meshing algorithm based on depth information detected in the physical environment, which is integrated (e.g., fused) to reconstruct the physical environment. The meshing algorithm (e.g., a double marching cube meshing algorithm, a Poisson meshing algorithm, a tetrahedron meshing algorithm, etc.) can be used to generate a mesh representing a room (e.g., the physical environment 105) and / or objects in the room (e.g., walls 130, doors 150, chairs 140, tables 142, etc.). In some embodiments, for 3D reconstruction using a mesh, in order to effectively reduce the amount of memory used during the reconstruction process, a voxel hashing method is used, in which the 3D space is divided into voxel blocks, which are referenced by a hash table using their 3D positions as keys. The voxel blocks are constructed only around the surface of the object, thereby freeing up memory that would otherwise be used to store empty space. The voxel hashing method was also faster than competing methods at the time, such as octree-based methods. In addition, it supported data flow between GPUs, where memory is usually limited, and CPUs, where memory is more abundant.

[0099] In one exemplary implementation, environment 400 includes an image synthesis pipeline that collects or acquires data of a physical environment (e.g., image data from an image source). Exemplary environment 400 is an example of collecting image data (e.g., light intensity data and depth data) for a plurality of image frames. The image source may include a depth camera 402 that collects depth data 404 of the physical environment, and a light intensity camera 406 (e.g., an RGB camera) that collects light intensity image data 408 (e.g., a sequence of RGB image frames).

[0100] The 3D representation unit 410 includes an integration unit 420 that is configured with instructions that can be executed by a processor to acquire image data (e.g., light intensity data 408, depth data 404, etc.) and integrate (e.g., fuse) the image data using one or more known techniques. For example, the image integration unit 420 receives depth image data 404 and intensity image data 408 from image sources (e.g., light intensity camera 406 and depth camera 402), and integrates the image data and generates 3D data 422. The 3D data 422 may include a dense 3D point cloud 424 (e.g., an imperfect depth map and camera orientation of multiple image frames around an object) that is sent to the semantic 3D unit 440. Gray points of different sizes in the 3D point cloud 424 represent different depth values ​​detected within the depth data. For example, the image integration unit 422 fuses an RGB image from a light intensity camera with a sparse depth map from a depth camera (e.g., a time-of-flight sensor) and other physical environment information sources to output a dense depth point cloud information. The 3D data 422 may also be voxelized, as represented by voxelized 3D point cloud 426 , where different shading on each voxel represents different depth values.

[0101] The 3D representation unit 410 also includes a semantic unit 430 configured with instructions that can be executed by the processor to obtain light intensity image data (e.g., light intensity data 408) and semantically segment wall structures (walls, doors, windows, etc.) and object types (e.g., tables, teapots, chairs, vases, etc.) using one or more known techniques. For example, the semantic unit 430 receives the intensity image data 408 from an image source (e.g., light intensity camera 406) and generates semantic segmentation data 432 (e.g., RGB-S data). For example, the semantic segmentation 434 shows Figure 1 In some implementations, the semantic unit 430 uses a machine learning model, wherein the semantic segmentation model can be configured to identify semantic labels of pixels or voxels of the image data. In some implementations, the machine learning model is a neural network (e.g., an artificial neural network), a decision tree, a support vector machine, a Bayesian network, etc.

[0102] The 3D representation unit 410 also includes a semantic 3D unit 440, which is configured with instructions that can be executed by the processor to obtain 3D data 422 (e.g., 3D point cloud data 424) from the integration unit 420 and semantic segmentation data 432 (e.g., RGB-S data) from the semantic unit 430, and generate a semantic 3D representation 445 using one or more techniques. For example, the semantic representation unit 440 generates a semantically labeled 3D point cloud 447 by collecting 3D point cloud data 424 and semantic segmentation 434 using a semantic 3D algorithm that fuses 3D data and semantic labels. In some specific implementations, each semantic label includes a confidence value. For example, a specific point can be labeled as an object (e.g., a table), and the data point will include x, y, z coordinates and a confidence value in decimal value (e.g., 0.9 represents a 90% confidence that the semantic label has correctly segmented the specific data point). In some specific implementations, a 3D reconstructed mesh can be generated as the semantic 3D representation 445.

[0103] Figure 5 is a flowchart representation of an exemplary method 500 for generating and displaying a real-time preview of a preliminary 2D floor plan of a physical environment based on a 3D representation of the physical environment according to some implementations. In some implementations, the method 500 is performed by a device (e.g., Figures 1 to 3 The method 500 may be performed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., a server 110 or device 120). Figure 1 and Figure 3 The method 500 is performed on a device 120 such as a head mounted display (HMD). In some implementations, the method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 500 is performed by a processor executing code stored in a non-transitory computer readable medium (e.g., a memory).

[0104] Method 500 is a process for creating a real-time preview of a preliminary floor plan that is displayed during a room scan (e.g., while walking around with a device such as a smart phone or tablet). To enable real-time preview of the preliminary floor plan, the preview can be generated (at least initially) differently than the final scanned floor plan (e.g., additional post-processing techniques for fine-tuning, improved accuracy for measurement data, etc.). For example, the real-time preview can use a less computationally intensive neural network, or provide an initial floor plan without fine-tuning (e.g., corner correction techniques). The use of 2D semantic data (e.g., for different layers of the room) can also facilitate making the preview determination valid for real-time display. According to some specific implementations, the preliminary floor plan creation process includes a 2D top view of the room based on individually identifying wall structures (e.g., wall edges, doors, and windows) and bounding boxes of detected objects (e.g., furniture, appliances, etc.). In addition or alternatively, the preliminary floor plan creation process for real-time preview and / or post-processing provides a 2D top view of the room based on identifying wall structures (wall edges), wherein the identified wall structures are based on a 2D representation of 3D semantic data encoded in multiple layers. Reference Figure 6 A real-time preview of the preliminary floor plan creation process of method 500 is shown.

[0105] At block 502, method 500 displays a real-time camera feed that includes a sequence of images of a physical environment. For example, as a user captures video while walking around a room to capture images of different parts of the room from multiple perspectives, these images are displayed in real time on a mobile device so that the user sees what he / she is capturing.

[0106] At block 504, the method 500 acquires a 3D representation of the physical environment, which is generated based on depth data and light intensity image data acquired during display of a real-time camera feed. The depth data may include pixel depth values ​​from a viewpoint and sensor position and orientation data. In some implementations, the depth data is acquired using one or more depth cameras. For example, the one or more depth cameras may acquire depth based on structured light (SL), passive stereo (PS), active stereo (AS), time of flight (ToF), etc. Various techniques may be applied to acquire depth image data to allocate each portion of the image (e.g., at the pixel level). For example, voxel data (e.g., a raster graphic on a 3D grid with values ​​of length, width, and depth) may also include multiple scalar values, such as opacity, color, and density. In some implementations, the depth data is acquired from a sensor or a 3D model of the image content. Some or all of the content of the image may be based on a real environment, such as a physical environment 105 surrounding the depicted device 120. The image sensor may capture an image of the physical environment 105 to be included in the image and depth information about the physical environment 105. In some implementations, a depth sensor (e.g., depth camera 402) on device 120 determines depth values ​​for voxels determined based on images captured by an image sensor on device 120. The physical environment 105 surrounding the user can be 3D modeled (e.g., 3D point cloud 424) based on the one or more values, and subsequent depths of objects depicted in subsequent images of the physical environment can be determined based on the model and camera position information (e.g., SLAM information).

[0107] At box 506, method 500 generates a real-time preview of a preliminary 2D floor plan of the physical environment based on the 3D representation of the physical environment. For example, a 2D top view of the preliminary floor plan of the physical environment 105 can be generated, which includes structures (e.g., walls, tables, doors, windows, etc.) identified in the room. In some specific implementations, the use of 2D semantic data (e.g., for different layers of the room) can also facilitate making the preview determination effective for real-time display. According to some specific implementations, the preliminary floor plan creation process includes a 2D top view of the room based on individually identifying wall structures (e.g., wall edges, doors and windows) and detecting bounding boxes of objects (e.g., furniture, appliances, etc.). In addition or alternatively, the preliminary floor plan creation process for real-time preview and / or post-processing provides a 2D top view of the room based on identifying wall structures (wall edges), wherein the identifying wall structures are based on a 2D representation of 3D semantic data encoded in multiple layers.

[0108] At block 508, method 500 displays a live preview of the preliminary 2D floor plan simultaneously with the live camera feed. For example, when the live camera feed is shown as the primary video, the preview may be displayed on a device (e.g., Figure 1 and Figure 3A picture-in-picture display is shown on a display of a device 120 of the user's computer system, and an image of the preliminary 2D floor plan is shown as the system builds the floor plan as the user acquires more sequences of image data (e.g., moves around the room). For example, while the user is viewing a live camera feed of a room environment (e.g., room environment 105), another viewing window with the preliminary 2D floor plan is overlaid on top of the live camera feed as it is generated (e.g., picture-in-picture (PIP)). Figure 6 Shows an overlay live preview display.

[0109] According to some specific implementations, method 500 also includes generating a final 2D floor plan of the physical environment based on the 3D representation, wherein generating the final 2D floor plan uses a different process from generating the real-time preview of the preliminary 2D floor plan. For example, the different process uses a more computationally intensive neural network with fine-tuning (e.g., corner correction), etc. In some specific implementations, the different process includes segmenting corners and small walls based on the 3D representation using a more computationally intensive neural network, generating a transitional 2D floor plan based on the segmented corners and small walls, determining a refinement scheme for the transitional 2D floor plan using a standardized algorithm, and generating the final 2D floor plan of the physical environment based on the determined refinement scheme for the transitional 2D floor plan.

[0110] According to some specific implementations, method 500 also includes generating an edge map by identifying walls in the physical environment based on the 3D representation, updating the edge map by identifying wall attributes (e.g., doors and windows) in the physical environment based on the 3D representation, updating the edge map by identifying objects in the physical environment based on the 3D representation, and generating a real-time preview of the preliminary 2D floor plan based on the updated edge map including the identified walls, the identified wall attributes, and the identified objects. In some specific implementations, generating a real-time preview of the 2D floor plan includes generating 2D semantic data for multiple horizontal layers of the physical environment based on the 3D representation, and generating the preliminary 2D floor plan using the 2D semantic data. For example, each layer provides x, y semantics for a range of z values, for example, for each x, y position in the z value range of 0-10, the first layer can be the most common semantic label.

[0111] According to some implementations, method 500 also includes generating an edge map by identifying walls, wherein this includes determining a parametric refinement line of the edge map using a line fitting algorithm, and updating the edge map based on the parametric refinement line. In some implementations, updating the edge map by identifying wall attributes includes determining the boundaries of the identified wall attributes using a wall attribute neural network and an image sequence fed by a real-time camera (e.g., RGB data of a transparent window), and generating the refined boundaries based on a 3D representation associated with the identified wall attributes using a polygon heuristic algorithm. In some implementations, updating the edge map by identifying objects includes generating a 3D bounding box corresponding to the identified object in the physical environment based on the 3D representation, and generating a 2D representation of the 3D bounding box (e.g., a furniture icon or a flat 2D bounding box).

[0112] Figure 6 6 is a system flow diagram of an exemplary environment 600 in which the system can generate and display a real-time preview of a preliminary 2D floor plan of a physical environment based on a 3D representation of the physical environment (e.g., a 3D point cloud, a 3D mesh reconstruction, a semantic 3D point cloud, etc.). In some implementations, the system flow of the exemplary environment 600 is performed on a device (e.g., Figures 1 to 3 The system flow of the exemplary environment 600 may be displayed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., Figure 1 and Figure 3 The system flow of the exemplary environment 600 is executed on a device 120 such as a head mounted display (HMD). In some implementations, the system flow of the exemplary environment 600 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 600 is executed on a processor that executes code stored in a non-transitory computer readable medium (e.g., a memory).

[0113] The system flow of the exemplary environment 600 collects the physical environment (e.g., Figure 1 406), a semantic 3D representation from the semantic 3D unit 440 (e.g., semantic 3D representation 445), and a real-time preview unit 610 (e.g., Figure 2 The real-time preview unit 244 and / or Figure 3The SLAM system may include a multi-dimensional (e.g., 3D) laser scanning and range measurement system that is independent of GPS and provides real-time simultaneous positioning and mapping. The SLAM system may generate and manage very accurate point cloud data generated by reflections from laser scans of objects in the environment. The movement of any point in the point cloud is accurately tracked over time, so that the SLAM system can use points in the point cloud as reference points for location, maintaining an accurate understanding of its position and orientation as it travels through the environment.

[0114] The floor plan real-time preview unit 610 includes an edge mapping unit 612, a line fitting unit 613, a wall attribute unit 614, a post / tuning unit 615, an object detection unit 616, a post / tuning unit 617, and a floor plan preview integration unit 618. The edge mapping unit 612 and the line fitting unit 613 are used to generate and refine edge maps based on the semantic 3D representation of the identified wall using one or more of the technologies disclosed herein. For example, the edge mapping unit 612 obtains the 3D data (e.g., the semantic 3D representation 445) of the identified semantically marked wall from the semantic 3D unit 440 and generates an initial 2D edge map of the identified wall, and the line fitting unit 613 generates a refined 2D edge map using a line fitting algorithm. The wall attribute unit 614 and the post / tuning unit 615 are used to generate and refine wall attribute boundaries based on the semantic 3D representation of the identified wall attributes (e.g., doors and windows) using one or more of the technologies disclosed herein. For example, the wall attribute unit 614 obtains light intensity image data (e.g., keyframes from the light intensity data 408) of the identified semantically labeled doors and windows, and generates 2D boundaries of the identified doors and windows. The line fitting unit 613 obtains the 3D data (e.g., the semantic 3D representation 445) of the identified semantically labeled doors and windows from the semantic 3D unit 440, and generates a refined boundary with associated depth data for each identified door and window using one or more post-processing and fine-tuning algorithms. The object detection unit 616 and the post / tuning unit 617 are used to generate and refine bounding boxes based on the semantic 3D representation of the identified object using one or more of the techniques disclosed herein. For example, the object detection unit 616 obtains the 3D data (e.g., the semantic 3D representation 445) of the identified semantically labeled object from the semantic 3D unit 440, and generates an initial bounding box of the identified object, and the post / tuning unit 617 generates a refined bounding box using one or more post-processing and fine-tuning algorithms.

[0115] As the floor plan preview integration unit 618 obtains the refined edge map from the line fitting unit 613, the refined boundaries from the post / tuning unit 614, and the refined bounding box from the post / tuning unit 617, the floor plan preview integration unit 618 iteratively generates and updates the preliminary 2D floor plan preview feed. For example, as the user scans the room with the device's camera, the acquired image data is continuously updating, so the edge map, wall attribute boundaries, and bounding boxes of the object can be continuously updated with each iteration of the updated image data. The floor plan preview unit 610 sends the preliminary 2D floor plan preview feed (e.g., preview 2D floor plan 630) and the real-time camera feed to the device display 312. The device display 312 can display the real-time view (e.g., light intensity image data 408) and a picture-in-picture (PIP) display 620 including the preview 2D floor plan 630. The preview 2D floor plan 630 includes edge map walls 632a, 632b, 632c (e.g., representing walls 134, 130, 132, respectively), a boundary 634a (e.g., representing door 150), a boundary 634b (e.g., representing window 152), a bounding box 636a (e.g., representing table 142), and a bounding box 636b (e.g., representing chair 140). In some implementations, standardized icons are used for identified objects (e.g., if a table is identified in the 3D representation data, a "table" icon is displayed instead of a bounding box or 2D box, such as Figure 6 636a).

[0116] Figure 7 700 is a flowchart representation of an exemplary method 700 for generating and displaying a 2D floor plan of a physical environment according to some specific implementations. In some specific implementations, the method 700 is performed by a device (e.g., Figures 1 to 3 The method 700 may be performed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., a server 110 or device 120). Figure 1 and Figure 3 The method 500 is performed on a device 120 such as a head mounted display (HMD). In some implementations, the method 500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 700 is performed by a processor executing code stored in a non-transitory computer readable medium (e.g., a memory).

[0117] Method 700 is a process of creating a floor plan of a physical space (e.g., physical environment 105). Method 700 provides a floor plan that includes a 2D top view of a room based on individually identifying wall structures (wall edges, doors, windows) in the room and generating bounding boxes of detected objects (e.g., furniture, appliances, etc.). Additionally or alternatively, the floor plan creation process provides a 2D top view of the room based on identifying wall structures (wall edges), wherein the identifying wall structures is based on a 2D representation of 3D semantic data encoded in multiple layers. Reference FIG. 8A to FIG. 8D The floor plan creation process of method 700 is shown.

[0118] At block 702, method 700 acquires a 3D representation of a physical environment, the 3D representation of the physical environment generated based on depth data and light intensity image data acquired during a scanning process. For example, a 3D point cloud may be generated based on depth camera information received simultaneously with the image during a room scan. In some implementations, the 3D representation is associated with 3D semantic data including a 3D point cloud, the 3D point cloud including semantic tags associated with at least a portion of the 3D points within the 3D point cloud (e.g., semantic 3D point cloud 447). Additionally, in some implementations, the semantic tags identify walls, wall attributes (e.g., doors and windows), objects, and divisions of objects of the physical environment.

[0119] At block 704, method 700 detects the location of a wall structure in a physical environment based on the 3D representation. For example, a wall may be identified by generating 2D semantic data (e.g., in layers), using the 2D semantic data to generate an edge map using a neural network, and determining vector parameters to normalize the edge map in a 3D normalized plan view. Wall attributes or wall properties (e.g., doors / windows) may be identified based on the RGB image and the depth data to generate a polygonal boundary. This technique for doors and windows provides advantages, particularly since the transparency of windows creates noise / errors in the depth data.

[0120] At block 706, method 700 generates bounding boxes corresponding to objects in the physical environment based on the 3D representation. For example, the 3D bounding box may provide the location, position (e.g., location and orientation), and shape of each piece of furniture and appliance in the room. The bounding box may be refined using RGB data and novel multi-network training techniques (e.g., 2-stage neural network fine-tuning for low precision / high recall and high precision / low recall).

[0121] At block 708, method 700 displays a 2D floor plan that provides a view (e.g., a top view) of the physical environment. In some implementations, the 2D floor plan is determined based on the location of the wall structure and bounding box corresponding to the object. For example, the 2D floor plan is displayed on a device (e.g., Figure 1 and Figure 3 The device 120 is displayed.

[0122] According to some specific implementations, method 700 also includes detecting the location of a wall structure in a physical environment based on the 3D representation, including identifying walls and wall attributes (e.g., doors and windows) of the physical environment from the wall structure based on the 3D representation, and generating an edge map of the identified walls and wall attributes based on the 3D representation, wherein the 2D plan is based on the generated edge map including the identified walls and the identified wall attributes. In some specific implementations, the exemplary method also involves using a neural network with higher computational density to segment out corners and small walls, updating the 2D plan based on the segmented corners and small walls, and determining a refined final 2D plan based on the updated 2D plan using a normalization algorithm. In some specific implementations, the exemplary method also involves using a wall structure neural network and light intensity image data (e.g., RGB data) acquired during the scanning process to determine the boundaries of the identified wall structure, and using a polygon heuristic algorithm to generate a refined boundary based on 3D semantic data associated with the identified wall attributes.

[0123] According to some implementations, the bounding boxes are refined bounding boxes, and the method 700 further involves generating a refined bounding box for the object by generating a proposed bounding box using a first neural network, and generating a refined bounding box by identifying features of the object using a second neural network (e.g., low precision / high recall to generate features of the object) and refining the proposed bounding box based on the identified features using a third neural network (e.g., high precision / low recall to refine the accuracy of the generated features and output a refined bounding box). In some implementations, the first neural network generates the proposed bounding box based on a 3D representation associated with the object. In some implementations, the second neural network identifies features of the object based on a 3D representation associated with the object and light intensity image data (e.g., RGB data) acquired during a scanning process. In some implementations, the third neural network is trained to refine the accuracy of the identified features from the second neural network, and outputs a refined bounding box based on the 3D representation associated with the object and light intensity image data (e.g., RGB data) acquired during a scanning process. In some implementations, a bounding box provides location information, position information (eg, location and orientation information), and shape information of an object in a physical environment.

[0124] Fig. 8A 800A, in which the system can generate and display a 2D plan view of a physical environment based on a 3D representation of the physical environment (e.g., a 3D point cloud, a 3D mesh reconstruction, a semantic 3D point cloud, etc.). In some implementations, the system flow of the exemplary environment 800A can be displayed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., Figure 1 and Figure 3The system flow of the exemplary environment 800A is executed on a device 120 such as a head mounted display (HMD). In some implementations, the system flow of the exemplary environment 800A is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 800A is executed on a processor that executes code stored in a non-transitory computer readable medium (e.g., a memory).

[0125] The system flow of the exemplary environment 800A collects the physical environment (e.g., Figure 1 406), a semantic 3D representation from the semantic 3D unit 440 (e.g., semantic 3D representation 445), and in the plan view finalization unit 850 (e.g., Figure 2 Floor plan unit 246 and / or Figure 3 Other sources of physical environment information (e.g., camera positioning information) at the real-time floor plan 346 of the present invention. Some specific implementations of the present disclosure may include a SLAM system (e.g., a SLAM unit 602).

[0126] The plan view unit 802 includes a wall structure unit 810, an object detection unit 840 (in Fig.8D ) and a plan finalization unit 850. The wall structure unit 810 includes a wall unit 820 (in Figure 8B ) and wall attribute unit 830 (shown in Figure 8C ). When the floor plan finalization unit 850 obtains the refined edge map from the wall unit 820, the refined boundary from the wall attribute unit 830, and the refined bounding box from the object detection unit 840, the floor plan finalization unit 850 generates 2D floor plan finalization data. The floor plan finalization unit 850 sends the 2D floor plan (e.g., 2D floor plan 860) to the device display (e.g., the display 312 or the device 120). The 2D floor plan 860 includes edge map walls 862a, 862b, 862c (e.g., representing walls 134, 130, 132, respectively), a boundary 864a (e.g., representing the door 150), a boundary 864b (e.g., representing the window 152), a bounding box 866a (e.g., representing the table 142), and a bounding box 866b (e.g., representing the chair 140).

[0127] In some implementations, the floor plan finalization unit 850 includes a normalization unit that refines the 2D floor plan using a standardized algorithm. For example, architectural floor plans used in the industry have common features or elements that meet standard floor plans, making reading floor plans easier and more efficient. Some standards include using generic icons to replace recognized objects, such as furniture, appliances, etc., instead of bounding boxes. The measurement data will still reflect the refined bounding box x, y, z measurements, but icons representing objects may be used.

[0128] In some embodiments, the floor plan finalization unit 850 includes a measurement unit to generate measurement data based on 3D representations of walls identified on the edge map, measurement data of boundaries of identified wall attributes, and measurement data of bounding boxes of identified objects using one or more processes further disclosed herein.

[0129] Figure 8B 800B, wherein the wall unit 820 can generate a refined edge map based on a 3D representation of the physical environment. In some implementations, the system flow of the exemplary environment 800B is a diagram of a system flow of an exemplary environment 800B. Figures 1 to 3 The system flow of the exemplary environment 800B is executed on a server 110 or device 120 such as a mobile device, a desktop computer, a laptop computer, or a server device. In some implementations, the system flow of the exemplary environment 800B is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 800B is executed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0130] The wall unit 820 includes an edge map neural network 822, a line fitting unit 826, and a small wall neural network 828. The system flow of the exemplary environment 800B begins with the edge map neural network 822 acquiring a semantic 3D representation (e.g., the semantic 3D representation 445) that includes 3D data of the identified wall. The edge map neural network 822 generates an initial edge map 823 of the identified wall and divides out corners 824a to 824g (referred to herein as corners 824). The edge map 823 is then refined by the line fitting unit 826 using a line fitting algorithm to generate a line fitting edge map 827. The line fitting edge map 827 is then further refined by the small wall neural network 828, which further divides and distinguishes each corner to generate a refined edge map 829. For example, corners 824a and 824e are initially identified as standard corners by the collected data, but the small wall neural network 828 is trained to identify corners that may actually be pillars or recessed corners so that the finalized floor plan should reflect accuracy and completeness. Additionally, corner 824d may actually be an open passage to an adjacent room, rather than a wall as initially indicated by the edge map. The refined edge map 829 is then sent to the floor plan finalization unit 850.

[0131] Figure 8C800C, in which the wall attribute unit 830 can generate a refined 2D boundary with associated depth data based on a light intensity image (e.g., a key RGB frame) and a 3D representation of a physical environment. In some implementations, the system flow of the exemplary environment 800C is performed on a device (e.g., Figures 1 to 3 The system flow of the exemplary environment 800C is executed on a server 110 or device 120 of the exemplary environment 800C, such as a mobile device, a desktop computer, a laptop computer, or a server device. In some specific implementations, the system flow of the exemplary environment 800C is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the system flow of the exemplary environment 800C is executed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0132] The wall attribute unit 830 includes a wall attribute neural network 832 and a post-processing unit 834. The system flow of the exemplary environment 800C begins with the wall attribute unit 830 acquiring a light intensity image (e.g., light intensity image data 408) at the wall attribute neural network 832, which generates initial boundaries 833a and boundaries 833b of the identified wall attributes (e.g., boundaries representing the door 150 and the window 152, respectively). The boundaries 833a, 833b are then refined by the post-processing unit 834, which obtains a semantic 3D representation of the 3D data including the identified wall attributes (e.g., the semantic 3D representation 445), and uses a polygon heuristic algorithm and generates refined 2D boundaries 835a, 835b with associated depth data. The refined 2D boundaries 835a, 835b are then sent to the floor plan finalization unit 850.

[0133] Fig.8D 800D, in which the object detection unit 840 can generate a refined bounding box for an associated identified object based on a 3D representation of the physical environment. In some implementations, the system flow of the exemplary environment 800D is performed on a device (e.g., Figures 1 to 3 The system flow of the exemplary environment 800D is executed on a server 110 or device 120 of the exemplary environment 800D, such as a mobile device, a desktop computer, a laptop computer, or a server device. In some specific implementations, the system flow of the exemplary environment 800D is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the system flow of the exemplary environment 800D is executed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0134] The object detection unit 840 includes an object detection neural network 842 and a fine-tuning unit 844, which includes a fine-tuning stage 1 neural network 846 and a fine-tuning stage 2 neural network 848. The system flow of the exemplary environment 800D begins with the object detection unit 840 acquiring a semantic 3D representation (e.g., semantic 3D representation 445) including 3D data of the identified object at the object detection neural network 842, which generates proposed bounding boxes 843a and 843b for the identified objects (e.g., table 142 and chair 140, respectively). The proposed bounding boxes 843a and 843b are then refined by the fine-tuning unit 844 using a two-stage neural network. The fine-tuning stage 1 neural network 846 acquires the semantic 3D representation data, the light intensity image data (e.g., light intensity image data 408), and the proposed bounding boxes 843a and 843b, and generates a stage 1 output. The fine-tuning stage 1 neural network 846 uses a neural network to identify features of the object using a low precision / high recall network to generate features of the object. The 3D data, light intensity image data, proposed bounding boxes 843a, 843b, and stage 1 output are taken by a fine-tuning stage 2 neural network 848, which generates refined bounding boxes using a high precision / low precision neural network to refine the accuracy of the generated features and output refined bounding boxes 845a and 845b (e.g., table 142 and chair 140, respectively). Fig.8D As shown in , the refined bounding boxes 845a and 845b are more accurate than the bounding boxes 843a and 843b, respectively. The refined bounding boxes 845a and 845b are then sent to the plan finalization unit 850.

[0135] Fig. 9 9 is a flowchart representation of an exemplary method 900 for generating and providing a floor plan of a physical environment based on generating an edge map using 2D semantic data according to some specific implementations. In some specific implementations, the method 900 is performed by a device (e.g., Figures 1 to 3 The method 900 may be performed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., a server 110 or device 120). Figure 1 and Figure 3 The method 900 is performed on a device 120 such as a head mounted display (HMD). In some implementations, the method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 900 is performed by a processor executing code stored in a non-transitory computer readable medium (e.g., a memory).

[0136] Method 900 is a process for creating a floor plan of a physical space (e.g., physical environment 105) based on 2D semantic data. Method 900 provides a floor plan that includes a 2D top view of a room based on generating multiple horizontal layers of 2D semantic data based on received semantic 3D semantic data. Fig.10 The floor plan creation process of method 900 is shown.

[0137] At box 902, method 900 obtains 3D semantic data of a physical environment, which is generated based on depth data and light intensity image data acquired during a scanning process. For example, a 3D point cloud can be generated based on depth camera information received simultaneously with the image during a room scan. In some specific implementations, the 3D semantic data includes a 3D point cloud that includes semantic tags associated with at least a portion of the 3D points within a 3D point cloud (e.g., semantic 3D point cloud 447) representation of the physical environment. In addition, in some specific implementations, the semantic tags identify walls, wall attributes (e.g., doors and windows), objects, and divisions of objects of the physical environment.

[0138] At block 904, method 900 generates multiple horizontal layers of 2D semantic data for the physical environment based on the 3D semantic data. For example, each layer provides x, y semantics for a range of z values, e.g., for each x, y position in a z value range of 0-10, the first layer may be the most common semantic label.

[0139] At block 906, method 900 provides a floor plan based on generating an edge map using the 2D semantic data, wherein the floor plan provides a view (e.g., a bird's-eye view) of the physical environment. In some implementations, generating the edge map may involve determining a parameter representation and / or vector parameters to normalize the edge map in a 3D normalized floor plan.

[0140] According to some specific implementations, method 900 also includes generating an edge map by identifying walls in the physical environment based on 2D semantic data of multiple horizontal layers, updating the edge map by identifying wall attributes (e.g., doors and windows) in the physical environment based on 3D semantic data, updating the edge map by identifying objects in the physical environment based on the 3D semantic data, and generating a floor plan based on the updated edge map including the identified walls, the identified wall attributes, and the identified objects.

[0141] According to some specific implementations, for method 900, the identified wall is a floor-to-ceiling wall (e.g., not a partition wall), wherein identifying the floor-to-ceiling wall based on the 2D semantic data of multiple horizontal layers includes identifying a floor of the physical environment having a lowest level of the multiple horizontal layers, identifying a ceiling of the physical environment having a highest level of the multiple horizontal layers, determining that the particular identified wall is not a floor-to-ceiling wall (e.g., a partition wall) based on the particular identified wall not satisfying a height threshold compared to the height of the ceiling, and updating the edge map by removing the particular identified wall from the edge map.

[0142] In some implementations, method 900 further includes generating an edge map by identifying walls, further including determining a parametric refinement line of the edge map using a line fitting algorithm, and updating the edge map based on the parametric refinement line. In some implementations, updating the edge map by identifying wall attributes includes determining a boundary of the identified wall attributes using a wall attribute neural network and a light intensity image (e.g., RGB data of a transparent window) acquired during a scanning process, and generating a refined boundary based on 3D semantic data associated with the identified wall attributes using a polygon heuristic algorithm.

[0143] In some implementations, method 900 also includes updating the edge map by identifying the object includes generating a 3D bounding box corresponding to the identified object in the physical environment based on the 3D semantic data, and generating a 2D representation of the 3D bounding box (e.g., a furniture icon or a flat 2D bounding box). In some implementations, the bounding boxes are refined bounding boxes, and generating the refined bounding boxes of the objects includes generating suggested bounding boxes using a first neural network, and generating the refined bounding boxes by identifying features of the object using a second neural network (e.g., low precision / high recall to generate features of the object) and refining the suggested bounding boxes based on the identified features using a third neural network (e.g., high precision / low recall to refine the accuracy of the generated features and output a refined bounding box).

[0144] Fig.10 is a system flow diagram of an exemplary environment 1000, wherein the system can generate and provide a 2D floor plan of a physical environment based on a 3D representation of the physical environment (e.g., a 3D point cloud, a 3D mesh reconstruction, a semantic 3D point cloud, etc.). In some implementations, the system flow of the exemplary environment 1000 is performed on a device (e.g., Figures 1 to 3 The system flow of the exemplary environment 1000 may be displayed on a device having a screen for displaying images and / or a screen for viewing stereoscopic images (e.g., Figure 1 and Figure 3The system flow of the exemplary environment 1000 is executed on a device 120 such as a head mounted display (HMD). In some implementations, the system flow of the exemplary environment 1000 is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 1000 is executed on a processor that executes code stored in a non-transitory computer readable medium (e.g., a memory).

[0145] The system flow of the exemplary environment 1000 collects the physical environment (e.g., Figure 1 The physical environment 105 may include image data (e.g., a real-time camera feed from the light intensity camera 406), a semantic 3D representation (e.g., semantic 3D representation 445) from the semantic 3D unit 440, and other sources of physical environment information at the floor plan unit 1010 (e.g., camera positioning information).

[0146] The plan view unit 1010 includes a semantic layer segmentation unit 1020, an edge mapping unit 1030, a line fitting unit 1032, a wall attribute unit 1040, a post / tuning unit 1042, an object detection unit 1050, a post / tuning unit 1052, and a plan view finalization unit 1060. The semantic layer segmentation unit 1020 obtains image data and a semantic 3D representation (e.g., a voxelized 3D point cloud 1022), and encodes the semantic confidence value and positioning data (e.g., global coordinates) of the 3D point cloud into a 3D semantic layer (e.g., a 3D semantic layer 1024). The semantic layer segmentation unit 1020 then generates a 2D representation (e.g., a 2D semantic layer 1026) for each 3D semantic layer. The semantic layer segmentation unit 1020 then generates a height map of the 2D semantic layer. For example, the 2D semantic height map 1028 may be used to determine whether a semantically identified wall is a floor-to-ceiling wall that should be included in the floor plan based on a comparison of an identified height threshold to an identified ceiling height, or whether a semantically identified wall does not reach the height of the ceiling (e.g., a partition wall), and then the system (e.g., the floor plan unit 1010) may determine not to include that particular wall in the edge map and associated floor plan. For example, if the wall height threshold is set to 90% of the ceiling height (e.g., for a 10-foot ceiling, there would be a 9-foot height threshold), and the height of the identified wall is determined to be 6 feet based on the 2D semantic layer, then the identified wall will be marked by the floor plan unit 1010 as a partition wall and will not be associated with the edge map. In some implementations, walls that do not meet the ceiling height (e.g., partition walls) may be designated as partition objects, and associated bounding boxes (e.g., the object detection unit 1050) may be generated using the techniques for object detection described herein.

[0147] The edge mapping unit 1030 and the line fitting unit 1032 are used to generate and refine edge maps based on the hierarchical 2D semantic layer 1028 using one or more of the techniques disclosed herein. For example, the edge mapping unit 612 obtains the encoded 3D data (e.g., 2D semantic layer 1028) of the identified semantically marked wall from the semantic 3D unit 440 and generates an initial 2D edge map of the identified wall, and the line fitting unit 1032 generates a refined 2D edge map using a line fitting algorithm. The wall attribute unit 1040 and the post / tuning unit 1042 are used to generate and refine wall attribute boundaries based on the 2D semantic layer 1028 of the identified wall attributes (e.g., doors and windows) using one or more of the techniques disclosed herein. For example, the wall attribute unit 1040 obtains light intensity image data (e.g., keyframes from the light intensity data 408) of the identified semantically marked doors and windows, and generates 2D boundaries of the identified doors and windows. The post / tuning unit 1042 obtains the 3D data (e.g., semantic 3D representation 445, 2D semantic layer 1028, etc.) of the identified semantically labeled doors and windows from the semantic 3D unit 440, and generates a refined boundary with associated depth data for each identified door and window using one or more post-processing and fine-tuning algorithms. The object detection unit 1050 and the post / tuning unit 1052 are used to generate and refine the bounding box based on the 2D semantic layer 1028 of the identified object using one or more techniques disclosed herein. For example, the object detection unit 1050 obtains the 3D data (e.g., semantic 3D representation 445, 2D semantic layer 1028, etc.) of the identified semantically labeled object from the semantic 3D unit 440 and generates an initial bounding box of the identified object, and the post / tuning unit 1052 generates a refined bounding box using one or more post-processing and fine-tuning algorithms.

[0148] When the plan finalization unit 1060 obtains the refined edge map from the edge mapping unit 1030 and the post / tuning unit 1032, obtains the refined boundary from the wall attribute unit 1040 and the post / tuning unit 1042, and obtains the refined bounding box from the object detection unit 1050 and the post / tuning unit 1052, the plan finalization unit 1060 generates 2D plan finalization data. The plan finalization unit 1060 sends the 2D plan (e.g., 2D plan 1062) to the device display (e.g., the display 312 or the device 120). The 2D plan 1062 includes the edge map wall (e.g., representing the Figure 1 of walls 134, 130, 132), wall attribute boundaries (e.g., Figure 1 150 and windows 152) and a bounding box (e.g., representing Figure 1 table 142 and chairs 140).

[0149] In some implementations, the floor plan finalization unit 1060 includes a normalization unit that refines the 2D floor plan using a standardized algorithm. For example, architectural floor plans used in the industry have common features or elements that meet standard floor plans, making reading floor plans easier and more efficient. Some standards include using generic icons to replace recognized objects, such as furniture, appliances, etc., instead of bounding boxes. The measurement data will still reflect the refined bounding box x, y, z measurements, but icons representing objects may be used.

[0150] In some embodiments, the floor plan finalization unit 1060 includes a measurement unit to generate measurement data based on a 3D representation of walls identified on the edge map, measurement data of boundaries of identified wall attributes, and measurement data of bounding boxes of identified objects using one or more processes further disclosed herein.

[0151] Fig.11 1 is a flowchart representation of an exemplary method 1100 for generating and providing measurements of wall structures and 3D bounding boxes associated with objects in a physical environment according to some implementations. In some implementations, the method 1100 is performed by a device (e.g., Figures 1 to 3 The method 1100 may be performed on a device (e.g., a server 110 or device 120) having a screen for displaying images and / or a screen for viewing stereoscopic images. Figure 1 and Figure 3 The method 1100 is performed on a device 120 such as a head mounted display (HMD). In some implementations, the method 1100 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1100 is performed by a processor executing code stored in a non-transitory computer readable medium (e.g., a memory).

[0152] Method 1100 is a process of creating measurement data of wall attributes (e.g., doors and windows) and objects of a physical space (e.g., physical environment 105). Method 1100 generates boundaries for identified wall structures (e.g., wall edges, doors, windows), and generates bounding boxes for detected objects in a room (e.g., furniture, appliances, etc.), and then provides measurement data based on the generated boundaries and bounding boxes. Reference FIG. 12 shows the measurement data creation process of method 1100.

[0153] At box 1102, method 1100 obtains a 3D representation of the physical environment generated based on the depth data obtained during the scanning process. For example, a 3D point cloud can be generated based on depth camera information received simultaneously with the image. In some embodiments, the 3D representation is associated with 3D semantic data. In some embodiments, the 3D representation is associated with 3D semantic data including a 3D point cloud, which includes semantic tags associated with at least a portion of the 3D points within the 3D point cloud (e.g., semantic 3D point cloud 447). In addition, in some embodiments, the semantic tags identify walls, wall attributes (e.g., doors and windows), objects, and divisions of objects of the physical environment. For example, the algorithm can be used for semantic segmentation and labeling of 3D point clouds of indoor scenes, where objects in the point cloud can have significant variations and complex configurations.

[0154] At block 1104, the method 1100 generates 2D boundaries of wall attributes in the physical environment based on the light intensity image of the physical environment. For example, all identified wall attributes (such as doors and windows) are analyzed relative to identified wall edges (e.g., floors) to generate 2D boundaries for each identified door and window based on the light intensity image (e.g., RGB). In an exemplary implementation, the light intensity image is utilized instead of depth data or a 3D representation (e.g., 3D point cloud 447) generated based on the depth data due to the transparency of the window that may provide inaccurate depth data.

[0155] At block 1106, method 1100 provides measurements of the wall attributes based on the 2D boundary and the 3D representation. After creating the boundary using only the light intensity image at block 1104, the system then generates measurement data using the 2D boundary and the 3D representation (e.g., 3D point cloud 447). For example, the 3D representation is used to determine the depth and / or width of a wall attribute such as a door or window given to a 2D polygonal shape associated with the wall attribute.

[0156] At block 1108, method 1100 generates 3D bounding boxes corresponding to objects in the physical environment based on the 3D representation. For example, the 3D bounding box may provide the location, position (e.g., location and orientation), and shape of each piece of furniture and appliance in the room. The bounding box may be refined using RGB data and a novel multi-network conditioning technique.

[0157] At block 1110, the method 1100 provides measurements of a 3D bounding box representing measurements of the corresponding object. For example, the length, width, and height of the bounding box correspond to the length, width, and height of the object.

[0158] According to some implementations, the bounding box is a refined bounding box, and method 1100 further includes generating a refined bounding box for the object by generating a proposed bounding box using a first neural network, and generating the refined bounding box by identifying features of the object using a second neural network (e.g., low precision / high recall to generate features of the object) and refining the proposed bounding box based on the identified features using a third neural network (e.g., high precision / low recall to refine the accuracy of the generated features and output the refined bounding box). In some implementations, the first neural network generates the proposed bounding box based on a 3D representation associated with the object. In some implementations, the second neural network identifies features of the object based on a 3D representation associated with the object and light intensity image data (e.g., RGB data) acquired during a scanning process. In some implementations, the third neural network is trained to refine the accuracy of the identified features from the second neural network, and outputs a refined bounding box based on the 3D representation associated with the object and light intensity image data (e.g., RGB data) acquired during a scanning process. In some implementations, a bounding box provides location information, position information (eg, location and orientation information), and shape information of an object in a physical environment.

[0159] According to some implementations, method 1100 further includes generating a refined boundary of the wall attribute based on the 3D semantic data associated with the wall attribute using a polygon heuristic algorithm. In some implementations, the wall attribute includes a door or a window.

[0160] According to some implementations, the measurements of the bounding box associated with the measurement of a particular wall attribute include the length, width, and height corresponding to the length, width, and height of the particular wall attribute. For example, the length, width, and height of a door. In some implementations, the measurements of the 3D bounding box for a particular object include the length, width, and height corresponding to the length, width, and height of the particular object. For example, the length, width, and height of a bounding box generated for a table or chair in a room.

[0161] Fig. 12A 12 is a system flow diagram of an exemplary environment 1200A, in which a wall attribute unit 1210 can generate a refined 2D boundary for wall attributes of a physical environment, and a floor plan measurement unit 1250 can provide measurements of the 2D boundary. In some implementations, the system flow of the exemplary environment 1200A is performed on a device (e.g., Figures 1 to 3The system flow of the exemplary environment 1200A is executed on a server 110 or device 120 of the exemplary environment 1200A, such as a mobile device, a desktop computer, a laptop computer, or a server device. In some implementations, the system flow of the exemplary environment 1200A is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the system flow of the exemplary environment 1200A is executed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0162] The wall attribute unit 1210 includes a wall attribute neural network 1212 and a post-processing unit 1214. The system flow of the exemplary environment 1200A begins with the wall attribute unit 1210 acquiring a light intensity image (e.g., light intensity image data 408) at the wall attribute neural network 1212, which generates initial boundaries 1213a and boundaries 1213b of the identified wall attributes (e.g., boundaries representing door 150 and window 152, respectively). The boundaries 1213a, 1213b are then refined by the post-processing unit 1214, which obtains a semantic 3D representation of the 3D data including the identified wall attributes (e.g., semantic 3D representation 445) and generates refined 2D boundaries 1215a, 1215b with associated depth data using a polygon heuristic algorithm. The refined 2D boundaries 1215a, 1215b are then sent to the floor plan measurement unit 1250 (e.g., Figure 2 The measurement unit 248 and / or Figure 3 The floor plan measurement unit 1250 obtains the semantic 3D representation of the associated boundaries (e.g., semantic 3D representation 445) and determines that the measurements of the boundaries associated with a particular wall attribute (e.g., boundaries 1252a and 1252b) include the length, width, and height corresponding to the length, width, and height of the particular wall attribute. For example, the length, width, and height of a door or window.

[0163] Fig. 12B 12 is a system flow diagram of an exemplary environment 1200B, in which an object detection unit 1220 can generate a refined bounding box for an associated identified object based on a 3D representation of a physical environment, and a floor plan measurement unit 1250 can provide measurements of the bounding box. In some implementations, the system flow of the exemplary environment 1200B is performed on a device (e.g., Figures 1 to 3 The system flow of the exemplary environment 1200B is executed on a server 110 or device 120 of the exemplary environment 1200B, such as a mobile device, a desktop computer, a laptop computer, or a server device. In some specific implementations, the system flow of the exemplary environment 1200B is executed on a processing logic component (including hardware, firmware, software, or a combination thereof). In some specific implementations, the system flow of the exemplary environment 1200B is executed on a processor that executes code stored in a non-transitory computer-readable medium (e.g., a memory).

[0164] The object detection unit 1220 includes an object detection neural network 1222 and an object fine-tuning unit 1230, which includes a fine-tuning stage 1 neural network 1232 and a fine-tuning stage 2 neural network 1234. The system flow of the exemplary environment 1200B begins with the object detection unit 1220 acquiring a semantic 3D representation (e.g., semantic 3D representation 445) including 3D data of the identified wall at the object detection neural network 1222, which generates proposed bounding boxes 1225a and 1225b for the identified objects (e.g., table 142 and chair 140, respectively). The proposed bounding boxes 1225a and 1225b are then refined by the object fine-tuning unit 1230 using a two-stage neural network. The fine-tuning stage 1 neural network 1232 acquires the semantic 3D representation data, the light intensity image data (e.g., light intensity image data 408), and the proposed bounding boxes 1225a and 1225b, and generates a stage 1 output. Fine-tune stage 1 neural network 1234 uses a neural network to identify features of an object using a low precision / high recall network to generate features of the object. The 3D data, light intensity image data, proposed bounding boxes 1225a, 1225b, and stage 1 output are taken by fine-tune stage 2 neural network 1234, which generates refined bounding boxes using a high precision / low recall neural network to refine the accuracy of the generated features and output refined bounding boxes 1235a and 1235b (e.g., table 142 and chair 140, respectively). Fig. 12B As shown in , the refined bounding boxes 1235a and 1235b are more accurate than the bounding boxes 1225a and 1225b, respectively. The refined bounding boxes 1235a and 1235b are then sent to the floor plan measurement unit 1250 (e.g., Figure 2 The measurement unit 248 and / or Figure 3 The floor plan measurement unit 1250 obtains the semantic 3D representation of the associated bounding box (e.g., semantic 3D representation 445) and determines that the measurements of each received bounding box associated with a particular object (e.g., bounding boxes 1262a and 1262b) include the length, width, and height corresponding to the length, width, and height of the particular object. For example, the length, width, and height of a table (e.g., table 142) or a chair (e.g., chair 140).

[0165] There are several implementations in which the bounding box measurements can be shown as overlaid in the composite image, for example by showing the edges and vertices of the bounding volume and / or by showing the surface of the bounding volume as partially transparent, so that the object and the bounding box are visible at the same time. In an exemplary embodiment, the spatial properties of the bounding box (e.g., length, height, and width) are automatically displayed to the user. Alternatively, the spatial properties are provided after the user interacts with the bounding box (e.g., selecting a bounding box icon or other selectable icon on the screen).

[0166] In some implementations, the image synthesis pipeline may include virtual content generated for an extended reality (XR) environment (e.g., placed in Figure 1 135 in the table 135). In some embodiments, the operating system 230, 330 includes built-in XR functionality, such as including an XR environment application or a viewer configured to be called from one or more applications 240, 340 to display the XR environment within a user interface. For example, the system described herein may include an XR unit configured with instructions executable by a processor to provide an XR environment, the XR environment including a description of a physical environment including real physical objects and virtual content. The XR unit may generate virtual depth data (e.g., a depth image of the virtual content) and virtual intensity data (e.g., a light intensity image (e.g., RGB) of the virtual content). For example, one of the applications 240 for the server 110 or the applications 340 for the device 120 may include an XR unit configured with instructions executable by a processor to provide an XR environment, the XR environment including a description of a physical environment including real objects or virtual objects. For example, virtual objects may be positioned based on detection, tracking, and representation of objects relative to each other in 3D space based on stored 3D models of real objects and virtual objects using one or more of the techniques disclosed herein.

[0167] Numerous specific details are set forth herein to provide a comprehensive understanding of the claimed subject matter. However, those skilled in the art will appreciate that the claimed subject matter may be practiced without these specific details. In other instances, methods, devices, or systems known to those of ordinary skill are not described in detail so as not to obscure the claimed subject matter.

[0168] Unless otherwise specifically noted, it should be understood that throughout the specification, discussions utilizing terms such as "process," "compute," "calculate," "determine," and "identify" refer to the actions or processes of a computing device, such as one or more computers or similar electronic computing devices, that manipulate or transform data represented as physical electronic or magnetic quantities within a memory, register, or other information storage device, transmission device, or display device of a computing platform.

[0169] The one or more systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components that provide results conditioned on one or more inputs. Suitable computing devices include multi-purpose microprocessor-based computer systems that access stored software that programs or configures the computing system from a general-purpose computing device to a dedicated computing device that implements one or more specific implementations of the subject matter of the present invention. Any suitable programming, scripting, or other type of language or combination of languages ​​may be used to implement the teachings contained herein in software for programming or configuring a computing device.

[0170] The specific implementation of the method disclosed herein can be performed in the operation of such a computing device. The order of the blocks presented in the above examples can be changed, for example, the blocks can be reordered, combined and / or divided into sub-blocks. Some blocks or processes can be executed in parallel.

[0171] The use of "suitable for" or "configured to" herein is meant to be open and inclusive language that does not exclude devices that are suitable for or configured to perform additional tasks or steps. In addition, the use of "based on" is meant to be open and inclusive, as a process, step, calculation, or other action "based on" one or more of the stated conditions or values ​​may in practice be based on additional conditions or values ​​beyond those stated. The headings, lists, and numbers included herein are for ease of explanation only and are not intended to be limiting.

[0172] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0173] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0174] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

[0175] The foregoing description and summary of the present invention should be understood to be illustrative and exemplary in every aspect, rather than restrictive, and the scope of the present invention disclosed herein is determined not only by the detailed description of the exemplary specific implementations, but by the full breadth allowed by the patent law. It should be understood that the specific implementations shown and described herein are only illustrative of the principles of the present invention, and that various modifications can be implemented by those skilled in the art without departing from the scope and essence of the present invention.

Claims

1. A method comprising: At an electronic device having a processor: Obtaining a three-dimensional 3D representation of the physical environment generated based on the depth data and the light intensity image data; determining a 3D position of the wall structure based on the 3D representation; determining a 3D position of a 3D bounding box corresponding to an object in the physical environment; Generate a floor plan based on: the 3D position of the wall structure; and the 3D position of the 3D bounding box corresponding to the object; and The plan view is provided for illustration purposes.

2. The method of claim 1 , wherein determining the 3D position of the wall structure in the physical environment based on the 3D representation comprises: identifying walls and wall properties of the physical environment from the wall structure based on the 3D representation; as well as An edge map of the identified walls is generated based on the 3D representation, wherein the plan view is based on the generated edge map, the generated edge map including the identified walls and the identified wall attributes.

3. The method according to claim 2, further comprising: Using a more computationally intensive neural network to segment corners and small walls based on the 3D representation; Generate a transitional plan based on the divided corners and small walls; determining a refinement for the transition plan using a standardized algorithm; as well as A final plan view of the physical environment is generated based on the determined refinements for the transition plan view.

4. The method according to claim 2, further comprising: determining a boundary for the wall structure using a wall structure neural network and the light intensity image data; as well as A polygon heuristic algorithm is used to generate a refined boundary based on 3D semantic data associated with the identified wall attributes.

5. The method of claim 1 , wherein the 3D bounding box is a refined 3D bounding box, wherein generating the refined 3D bounding box for the identified object comprises: Generate a 3D bounding box for the proposal using the first neural network; as well as A refined 3D bounding box is generated by identifying features of the identified object using a second neural network and refining the proposed 3D bounding box based on the identified features using a third neural network.

6. The method of claim 5, wherein the first neural network generates the proposed 3D bounding box based on the 3D representation associated with the identified object.

7. The method of claim 5, wherein the second neural network identifies features of the identified object based on the 3D representation and light intensity image data associated with the identified object.

8. The method of claim 5, wherein the third neural network is trained to refine the accuracy of the identified features from the second neural network and output a refined 3D bounding box based on the 3D representation and light intensity image data associated with the identified object.

9. The method of claim 1, wherein the 3D bounding box provides position information, pose information, and shape information of the object in the physical environment.

10. The method of claim 1, wherein the 3D representation is associated with 3D semantic data comprising a 3D point cloud, the 3D point cloud comprising semantic labels associated with at least a portion of 3D points within the 3D point cloud.

11. The method of claim 10, wherein the semantic tags identify walls, wall structures, objects, and divisions of the identified objects of the physical environment.

12. A device comprising: a non-transitory computer-readable storage medium; as well as One or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium includes program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising: Obtain a three-dimensional 3D representation of the physical environment generated from depth data and light intensity image data determining a 3D position of the wall structure based on the 3D representation; determining a 3D position of a 3D bounding box corresponding to an object in the physical environment; Generate a floor plan based on: the 3D position of the wall structure; and the 3D position of the 3D bounding box corresponding to the object; and The plan view is provided for illustration purposes.

13. The apparatus of claim 12, wherein determining the 3D position of the wall structure in the physical environment based on the 3D representation comprises: identifying walls and wall properties of the physical environment from the wall structure based on the 3D representation; as well as An edge map of the identified walls is generated based on the 3D representation, wherein the plan view is based on the generated edge map, the generated edge map including the identified walls and the identified wall attributes.

14. The apparatus of claim 13, wherein the operations further comprise: Using a more computationally intensive neural network to segment corners and small walls based on the 3D representation; Generate a transitional plan based on the divided corners and small walls; determining a refinement for the transition plan using a standardized algorithm; as well as A final plan view of the physical environment is generated based on the determined refinements for the transition plan view.

15. The apparatus of claim 13, wherein the operations further comprise: determining a boundary for the wall structure using a wall structure neural network and the light intensity image data; as well as A polygon heuristic algorithm is used to generate a refined boundary based on 3D semantic data associated with the identified wall attributes.

16. The apparatus of claim 12, wherein the 3D bounding box is a refined 3D bounding box, wherein generating the refined 3D bounding box for the identified object comprises: Generate a 3D bounding box for the proposal using the first neural network; as well as A refined 3D bounding box is generated by identifying features of the identified object using a second neural network and refining the proposed 3D bounding box based on the identified features using a third neural network.

17. The apparatus of claim 12, wherein the 3D representation is associated with 3D semantic data comprising a 3D point cloud, the 3D point cloud comprising semantic labels associated with at least a portion of 3D points within the 3D point cloud.

18. A non-transitory computer-readable storage medium storing program instructions executable on a device to perform operations comprising: Obtaining a three-dimensional 3D representation of the physical environment generated based on the depth data and the light intensity image data; determining a 3D position of the wall structure based on the 3D representation; determining a 3D position of a 3D bounding box corresponding to an object in the physical environment; Generate a floor plan based on: The 3D position of the wall structure; and the 3D position of the 3D bounding box corresponding to the object; and The plan view is provided for illustration purposes.

19. The non-transitory computer-readable storage medium of claim 18, wherein determining the 3D position of the wall structure in the physical environment based on the 3D representation comprises: identifying walls and wall properties of the physical environment from the wall structure based on the 3D representation; as well as An edge map of the identified walls is generated based on the 3D representation, wherein the plan view is based on the generated edge map, the generated edge map including the identified walls and the identified wall attributes.

20. The non-transitory computer readable storage medium of claim 19, wherein the operations further comprise: Using a more computationally intensive neural network to segment corners and small walls based on the 3D representation; Generate a transitional plan based on the divided corners and small walls; determining a refinement for the transition plan using a standardized algorithm; as well as A final plan view of the physical environment is generated based on the determined refinements for the transition plan view.

21. A method comprising: At an electronic device having a processor: acquiring sensor data of a physical environment including an object; Determining a three-dimensional 3D position of a wall structure and a 3D position of a bounding box corresponding to the object based on the sensor data; generating a plan view based on the 3D position of the wall structure and the 3D position of the bounding box; as well as The plan view is provided for illustration purposes.

22. A method comprising: At an electronic device having a processor: Acquire three-dimensional 3D semantic data of the physical environment generated based on the depth data and light intensity image data acquired during the scanning process; encoding the 3D semantic data into a 3D semantic layer; segmenting the 3D semantic data to separate the 3D semantic layer to provide a two-dimensional 2D semantic representation of each of a plurality of horizontal layers representing the physical environment; and A floor plan is provided based on generating an edge map using the 2D semantic representation, the floor plan providing a view of the physical environment.

23. A method comprising: At an electronic device having a processor: displaying a live camera feed comprising a sequence of images of a physical environment; acquiring a three-dimensional (3D) representation of a physical environment generated based on depth data and light intensity image data of the physical environment obtained during display of the real-time camera feed; as well as Generating a final plan view of the physical environment based on the 3D representation of the physical environment, wherein generating the final plan view comprises: identifying a wall in the physical environment based on the 3D representation; identifying wall properties in the physical environment based on the 3D representation; generating a first plan view based on the identified walls and the identified wall attributes; Refining the identified walls and the identified wall properties; identifying objects in the physical environment based on the 3D representation and generating 3D bounding boxes corresponding to the identified objects; and A second plan view is generated based on the refined identified walls, the refined identified wall attributes, and the 3D bounding boxes associated with the identified objects.

Images