System and method for detecting objects and interactive 3D models

Abstract

The system includes a processor and a memory, the memory including instructions to control the processor to receive an image representing an interior of a physical environment, use a neural network for object recognition to identify an object in the image associated with a location relative to the physical environment, use the neural network for object recognition to identify another object in another image, determine whether multiple objects in the multiple images are located near or in similar locations based on location information associated with the multiple objects, and if the multiple objects are located near or in similar locations, determine that the multiple objects are a single object instance, store the similar location associated with the single object, display an interactive walk-through visualization of a 3D model of the physical environment including the single object, receive a request for a location of the object through the interactive walk-through visualization, and provide the similar location of the single object for display in the interactive walk-through visualization.

Description

[Technical field]

[0001] Embodiments of the invention(s) relate to interactive 3D visualization that provides information about a 3D modeled environment, and more particularly to an automated system that detects objects, collects object information, and interactively responds to user queries about the 3D model and the objects contained within the 3D model. [Background technology]

[0002] Three-dimensional (3D) visualization and walkthroughs typically allow a user to view and / or interact with a 3D model of a given environment. 3D model visualization of physical environments, such as homes, is becoming commonplace. In one example, a user can access a 3D model through a website, such as a real estate website, and walkthrough the 3D model of the home. However, these 3D models are limited to walkthroughs and certain graphical elements that allow the user to move through the space. The 3D models do not identify objects within the model, nor do they allow the user to request information about the physical environment. Summary of the Invention [Means for solving the problem]

[0003] An example system includes one or more processors and a memory, the memory including instructions for controlling the one or more processors to receive a plurality of images representing an interior of a physical environment, use a neural network for object recognition to identify a first object in a first image of the plurality of images associated with location information of the first object indicative of a location relative to the physical environment, use the neural network for object recognition to identify a second object in a second image of the plurality of images, determine whether the first object in the first image and the second object in the second image are located near or in similar locations in the physical environment based on the location information associated with the first image and the location information of the second object associated with the second image, determine if the first object and the second object are located near or in similar locations in the physical environment if the first object and the second object are located near or in similar locations, determine that the first object and the second object are a single object instance, store the similar location associated with the single object, display an interactive walk-through visualization of the 3D model of the physical environment including the single object, receive an object information request through the interactive walk-through visualization for a location of the object, and provide the similar location of the single object for display in the interactive walk-through visualization.

[0004] In various embodiments, the instructions further control the one or more processors to determine whether the first object and the second object are of the same object type. In one example, if the first object and the second object are located near or in a similar location and the first object is of the same object type as the second object, the first object and the second object are a single object instance. In another example, identifying the first object using a neural network for object recognition further includes categorizing the first object and identifying an object type of the first object using the neural network. In one exemplary system, the instructions further control the one or more processors to determine location information of the first object and location information of the second object based on a mesh of the 3D model. In some embodiments, the first image includes a position and orientation of an image capture device that captured the first image within the physical environment, and the instructions further control the one or more processors to determine location information of the first object based on a position and orientation of the image capture device that captured the first image.

[0005] In one exemplary system, the second image includes a position and orientation of an image capture device that captured the second image within the physical environment, and the instructions further control the one or more processors to determine location information for the second object based on the position and orientation of the image capture device that captured the second image. In one example, the first image is associated with depth data of the physical environment, the depth data generated by a depth data device, and the location of the first object is determined based on the first object shown in the first image and the associated associated depth data. The system of claim 8, wherein the depth data device is a LiDAR.

[0006] An exemplary non-transitory computer readable medium comprises executable instructions, the executable instructions being executable by one or more processors to perform a method, the method including: receiving a plurality of images depicting an interior of a physical environment; identifying a first object in a first image of the plurality of images using a neural network for object recognition, the first object being associated with first object location information, the first object location information indicating a location relative to the physical environment; and identifying a second object in a second image of the plurality of images using a neural network for object recognition, based on the location information associated with the first image and the location information of the second object associated with the second image. The method includes determining whether a first object in the first image and a second object in the second image are located near or at similar locations in the physical environment, and if the first object and the second object are located near or at similar locations, determining that the first object and the second object are a single object instance; storing the similar locations associated with the single object; displaying an interactive walk-through visualization of a 3D model of the physical environment, the 3D model including the single object; receiving an object information request for locations of the objects through the interactive walk-through visualization; and providing the similar locations of the single object for display in the interactive walk-through visualization.

[0007] An example method includes receiving a plurality of images depicting an interior of a physical environment; identifying a first object in a first image of the plurality of images using a neural network for object recognition, the first object being associated with first object location information, the first object location information indicating a location relative to the physical environment; identifying a second object in a second image of the plurality of images using a neural network for object recognition; and determining, based on the location information associated with the first image and the location information associated with the second image, that the first object in the first image and the second object in the second image are in a similar location in the physical environment. the first object and the second object are located near or in a similar location; determining that the first object and the second object are a single object instance if the first object and the second object are located near or in a similar location; storing the similar location associated with the single object; displaying an interactive walk-through visualization of a 3D model of the physical environment, the 3D model including the single object; receiving an object information request through the interactive walk-through visualization for the location of the object; and providing the similar location of the single object for display in the interactive walk-through visualization.

[0008] Another example system includes one or more processors and a memory, the memory including instructions to control the one or more processors to identify a first view of a first object in the first two-dimensional image, the first view of a first object in the first two-dimensional image, the first set of pixels associated with a first position and a first orientation of the image capture device when the image capture device captured the first two-dimensional image, identify a second view of a second object in the second two-dimensional image, the second view of a second object in the second two-dimensional image, the second set of pixels associated with a second position and a second orientation of the image capture device when the image capture device captured the second two-dimensional image, map the first set of pixels to a position in a 3D model that is a representation of the physical environment based on the first position and first orientation of the image capture device, determine that the first object and the second object are the same physical object based on the mapping of the second set of pixels, the second position, and the first set of pixels in the second two-dimensional image, and represent the same physical object in the 3D model of the physical environment using at least a portion of the first set of pixels and the second set of pixels.

[0009] In some embodiments, determining that the first object and the second object are the same physical object is also based on a depth map of the physical environment. In one example, a first object location of the same physical object in the 3D model is determined based on a depth map of the physical environment. In one example, the first set of pixels is a first set of surface elements and the second set of pixels is a second set of surface elements. In various embodiments, the method further includes identifying an object type of the same physical object based at least in part on the first set of pixels. In some embodiments, identifying an object type of the same physical object is further based at least in part on the second set of pixels using a CNN to categorize the same physical object. In another example, the exemplary system further includes generating a first label associated with an object type and a location of the same physical object in the 3D model. The exemplary system further includes identifying a third view of the third object in the third two-dimensional image, the third set of pixels associated with a third position and a third orientation of the image capture device when the image capture device captured the third two-dimensional object, and determining that the third object is a different physical object from the first object and the second object based on the mapping of the third position and the first set of pixels. One exemplary system further includes identifying an object type of the third object, the object type of the third object being the object type of the first object. In various embodiments, the exemplary system further includes generating a second label associated with the third object and increasing a total number of object types indicative of the number of object types represented in the 3D model. Another example system further includes storing the first label and the second label in a data structure, receiving a query regarding a location of an object of the object type in the 3D model, and responding to the query based on information in the data structure including the first label and the second label, wherein the first object location of the first object in the 3D model and the second object location of a third object in the 3D model, the first object location of the same physical object, and the second object location of the third object are based on one or more depth maps of the physical environment. [Brief description of the drawings]

[0010] [Figure 1]FIG. 1 is a block diagram of an example environment capable of receiving digital images of a physical environment, automatically detecting objects shown in the digital images, and providing information about the detected objects. [Diagram 2] FIG. 1 is a block diagram of an object detection and location system in accordance with some embodiments. [Diagram 3] 1 is a flowchart of an object detection process according to some embodiments. [Figure 4] FIG. 13 is an example of an exemplary user interface for identifying locations of multiple detected objects of a similar type, according to some embodiments. [Diagram 5] 1 is a flowchart of a multiple capture position approach to the object detection process, according to some embodiments. [Figure 6] 1 is a flowchart for identifying the same or different object instances in two-dimensional images based on the position of the image capture device that captured the images in some embodiments. [Figure 7] FIG. 1 is an illustration of an example of image collection of the same physical space viewed by multiple image capture devices, where two-dimensional object detection is highlighted. [Figure 8] FIG. 1 is an illustration of an example of image collection of the same physical space viewed by multiple image capture devices, where two-dimensional object detection is highlighted. [Figure 9] FIG. 1 is an illustration of an example of image collection of the same physical space viewed by multiple image capture devices, where two-dimensional object detection is highlighted. [Figure 10] FIG. 1 is an illustration of an example of image collection of the same physical space viewed by multiple image capture devices, where two-dimensional object detection is highlighted. [Figure 11] FIG. 11 is a merge of the image collections of FIGS. 7-10, showing the location of the image capture device and the location of the detected two-dimensional object. [Figure 12] 1 is a flowchart for collecting information and providing interaction with a 3D model in some embodiments. [Figure 13] 1 is a flowchart of a mesh approach to object detection process according to some embodiments. [Figure 14]FIG. 1 is a schematic diagram of one implementation of an object detection and location system. [Figure 15A] 1 is a diagram of mesh components that make up an object in a physical environment, according to some embodiments. [Figure 15B] 1 is a diagram of the location, size and shape of two objects recognized in a physical environment, according to some embodiments. [Figure 16A] 1 is a diagram of mesh components that make up an object in a physical environment, according to some embodiments. [Figure 16B] FIG. 2 is a diagram of the location, size and shape of objects identified in a physical environment and highlighted in different colors, according to some embodiments. [Figure 17A] 1 is a diagram of locations of the same object detected from different viewpoints, according to some embodiments. [Figure 17B] 1 is a diagram of locations of the same object detected from different viewpoints, according to some embodiments. [Figure 17C] 1 is a diagram of locations of the same object detected from different viewpoints, according to some embodiments. [Figure 18] 1 is a flowchart of a depth estimation approach to an object detection process according to some embodiments. [Figure 19] FIG. 2 is an example of a 2D color image and a corresponding 2D depth map. [Figure 20] FIG. 1 is a diagram of an exemplary user interface showing identified objects by tags, where the objects correspond to three-dimensional points in physical space, according to some embodiments. [Figure 21] 1 is a flowchart of an object detection process according to some embodiments. [Figure 22] FIG. 1 is a block diagram of an exemplary digital device according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Various embodiments described herein enable the collection of information about various objects within a 3D model. A user may request information about a 3D model (e.g., through a visualization or walkthrough and / or interaction with an API) (e.g., by providing a query). In response, various systems discussed herein may extract relevant information about objects within the 3D model and reply to the request by providing text, images, and / or changes to the user's 3D model visualization to provide the information.

[0012] To enable collection of object information (e.g., object location in the 3D model, object type, number of objects of the same object type, etc.), it is necessary to identify objects that appear in the 3D model and collect information about these objects. Traditionally, identification of objects shown in images is performed manually. For example, a person may look at one image at a time and manually tag and / or identify the objects. A person may also manually provide descriptive metadata for the objects.

[0013] Unfortunately, manual image tagging is time consuming, expensive, and error prone, especially when the set of images to be tagged is large. Thus, as the number of images increases and / or even as the number of items to be tagged within an image increases, manual image tagging becomes unscalable.

[0014] To avoid the human effort involved in manual tagging, limited techniques have been developed to automatically recognize and tag digital images with certain information. For example, digital cameras may automatically store several types of information with each image, such as the time, date, and GPS coordinates at which the image was captured.

[0015] Various embodiments described herein include systems and methods that automatically identify objects that appear or will appear in a 3D model, classify (e.g., tag) the objects, identify the location of the objects in the 3D model, collect information about the objects, and store the associated information. The system may provide an interactive visualization of the 3D model (e.g., a walk-through interface that allows a user to walk through the 3D model) and allow a user to request (e.g., through a query field or interactive GUI elements) more information about the objects in the visualization (e.g., how many outlets are in the house?). The system may extract the collected information about the identified objects and provide a response to the request (e.g., through text, visual elements, etc.).

[0016] It will be appreciated that there are many ways to identify objects in an image (e.g., using multiple images, using depth information, or using a mesh) and many ways to collect object metadata (e.g., object type, number of object types per floor or in a building, location of one or more objects, etc.).

[0017] Various systems and methods described herein enable a scalable system to automatically detect objects, distinguish objects from one another (e.g., identifying different chairs on different floors), and collect location and other information for each object. Performing these steps manually can be error-prone, slow, and computationally inefficient.

[0018] 1 illustrates a block diagram of an exemplary environment 100. Environment 100 includes a communications network 102, (individually) environmental images 104A and 104B (collectively environmental images 104), (individually) image capture devices 106A and 106B (collectively image capture devices 106), a building 108, a residence 110, an object detection and location system 112, a user system 114, and a model data store 116.

[0019] In this example, the object detection and location system 112 may provide or enable a visualization and / or a 3D model to enable a user to conduct a walk-through of the modeled environment. The object detection and location system 112 may provide an interactive visualization and / or an option to enable a user to request information about one or more objects shown in the 3D model. The object detection and location system 112 may extract the information and provide a response. In various embodiments, the object detection and location system 112 may automatically identify objects from the 2D images and / or 3D models and collect information associated with the objects (e.g., identify object type, number of objects of a particular type shown in the 3D model or in a portion of the 3D model, location of the object(s) in the modeled environment, etc.).

[0020] In some embodiments, the communication network 102 represents one or more computer networks (e.g., a LAN, a WAN, etc.). The communication network 102 may provide communication between or among environments such as the image capture devices 106, the object detection and location system 112, the user systems 114, the model data store 116, etc. In some implementations, the communication network 102 comprises computer devices, routers, cables, equipment, and / or other network topologies. In some embodiments, the communication network 102 may be wired and / or wireless. In various embodiments, the communication network 102 may comprise one or more networks, which may be the Internet, public, private, IP-based, non-IP-based, etc.

[0021] In various embodiments, environmental image 104A includes digital images of a physical environment, such as the interior of building 108. These images may be captured by placing one or more image capture devices 106A at various locations within the building 108. In various embodiments, environmental image 104A includes digital images of the exterior of building 108. Environmental image 104A may show a sufficient amount of the interior of building 108 (e.g., living spaces on all floors) such that the images may serve as the basis for generation of a 3D model of the interior of building 108.

[0022] Digital images or video captured by image capture device(s) 106A may be transmitted to object detection and location system 112. Image capture device(s) 106 may transmit digital images to object detection and location system 112. In various embodiments, digital images of the interior and / or exterior of the identified location may be transmitted to model data store 116. Alternatively, image capture device(s) 106 may wirelessly couple to a smart device, which may provide the digital images to object detection and location system 112. In some embodiments, the images may be downloaded to a card or other medium and later uploaded to object detection and location system 112.

[0023] Image capture device 106 may include sensors and / or software that identify a position and / or orientation. In various embodiments, image capture device 106 may associate a position and / or orientation of image capture device 106 with one or more images captured by image capture device 106 at that position and / or orientation. In one example, the position of image capture device 106 may be provided by a GPS sensor that provides GPS coordinates, or any other system that aids in location determination. The orientation of the image capture device may be identified based on the position of the image capture device and the field of view from the lens of the image capture device. The orientation and / or position of image capture device 106 may be determined or identified in any of a number of ways.

[0024] In some embodiments, image capture device 106 is a complementary metal oxide semiconductor (CMOS) image sensor (e.g., Sony IMX283, approximately 20 megapixel CMOS MIPI sensor with NVidia Jetson Nano SOM). In various embodiments, image capture device is a charge-coupled device (CCD). In one example, image capture device is a red-green-blue (RGB) sensor. In one embodiment, image capture device 106 is an infrared (IR) sensor. Image capture device 106 may include a lens assembly that provides the image capture device with a wide field of view.

[0025] In some embodiments, the image capture device 106 may include a depth sensor (such as LiDAR, SPAD, or structured light) that obtains depth data. Depth data may be defined as the distance from the image capture device to a point in the physical environment that is indicated in a pixel of an image captured by the image capture device. Alternatively, in other embodiments, depth data may be obtained using multiple image sensors, such as stereo assisted imaging, where multiple image sensors are offset by a predetermined distance. These multiple image sensors may capture approximately the same physical environment, but with slight offsets. Digital images captured by these multiple image sensors may be utilized by a processor to generate or enhance the illusion of depth in the form of a three-dimensional image.

[0026] In yet other embodiments, the depth data may be captured by a LiDAR device (not shown in FIG. 1 ) that is separate from image capture device 106. It will be understood that depth data from a LiDAR sensor (e.g., part of image capture device 106 or separate from image capture device 106) may not be available.

[0027] In various embodiments, the depth data defines depth and / or location information about the physical environment being scanned. The depth data may relate to the locations of objects, walls, floors, ceilings, etc. that may be the subject of images captured by the image capture device(s).

[0028] In some embodiments, the object detection and localization system 112 may receive two-dimensional images of any number of physical environments from the image capture device(s) 106.

[0029] The object detection and localization system 112 may utilize various methods or techniques to identify and categorize objects shown in the two-dimensional images. The method or technique used by the object detection and localization system 112 may depend on the type of two-dimensional image received by the object detection and localization system 112. The various methods may include, but are not limited to, identifying objects based in part on the orientation and position of the image capture device(s) 106 that captured the image, using depth data (e.g., depth data generated from the image or LiDAR), or from a mesh.

[0030] In one example, the object detection and localization system 112 receives two-dimensional images (and optionally depth data) of the physical environment from the image capture device 106. In this approach, objects are identified or detected in two dimensions in a subset of the two-dimensional images (e.g., through the use of a neural network such as an R Mask CNN). The objects (e.g., pixels associated with the identified objects) may be mapped to three-dimensional surfaces or locations in the physical environment. A set of images of a particular identified object may be identified based on this mapping, and once a set of images of a particular object is identified, image information (e.g., pixels) of various views of the same physical object may be used in a 3D model visualization to show the object (e.g., at one or more viewpoint orientations).

[0031] In some embodiments, the object detection and localization system 112 trains a neural network or uses a trained neural network to recognize specific object types shown in the two-dimensional images. Once trained, the neural network may identify objects from images captured by one or more image capture devices 106. An object type is a specific type of object or one type of object. For example, an object type may be "chair," "table," "fireplace," "outlet," "toilet," "shower," "sink," etc. An object instance is a specific object of that object type. For example, an individual chair located in front of a fireplace in a room is an instance of a chair and has the object type "chair." Different individual chairs located on different floors are different instances of chair (i.e., there are two instances of chair in this example), and although both chairs are of the object type "chair," the chairs may be fundamentally different from one another (e.g., one may be a kitchen table chair while the other may be a recliner chair).

[0032] In some embodiments, the neural network may, for example, use instance segmentation to identify the location of every pixel of every object in an image (e.g., using Mask R-CNN or any R-CNN method). For each image, the object detection and localization system 112 may generate region proposals, classify the region proposals, generate segmentation masks, and identify specific objects for identification (e.g., tagging and / or labeling), counting, and / or localization.

[0033] In various embodiments, the object detection and localization system 112 provides (e.g., streams and / or outputs) all or a portion of the 3D model to a digital device (e.g., a user system 114 or a remote website). A user can utilize the user system 114 to view all or a portion of the 3D model (such as the exemplary two-dimensional panoramic image 1900 of FIG. 19).

[0034] In some embodiments, the object detection and localization system 112 may provide a visual representation of an area of ​​a building floor plan to the user system 114 (i.e., a 3D model visualization). Figure 4 is an example user interface 400. A user of the user system 114 can interact with an area 410 to view and move around parts of the building. In some embodiments, the user interacts with a list 420 that provides multiple instances of objects in a given object type (i.e., a category of objects).

[0035] In various embodiments, the exemplary user interface 400 provides an interactive walkthrough of a 3D model (e.g., a walkthrough of a 3D model of a physical environment). In the example of FIG. 4, a list 420 lists identified outlets located in the 3D model. A user may interact with one particular outlet, such as entrance 422. In response, the user interface 400 may provide the location of the particular outlet in a floor plan of the building.

[0036] In some embodiments, the exemplary user interface 400 may present a dollhouse view that is a walk-through view of the 3D model. In response to a user query, the exemplary user interface may extract information and / or provide a response to the user. In some embodiments, the visualization module 218 may present a location within the 3D model and present objects of an object type indicated by the query. For example, in FIG. 4, the visualization module 218 may highlight the location of each outlet on a 3D map, mark the outlets, annotate the outlets, etc., in response to a query. It will be appreciated that the visualization module 218 may present objects of interest in any number of ways (e.g., pulsing, glowing, highlighting, arrows, etc.).

[0037] In some embodiments, the user system 114 is a digital device that may communicate with other digital devices and systems. A digital device is any device that has a processor and memory. In some embodiments, the user system 114 may be or include one or more mobile devices (e.g., a smartphone, a mobile phone, a smartwatch, a tablet computer, etc.), a desktop computer, a laptop computer, etc. In some embodiments, a user may interact with the user system 114 and communicate with the object detection and location system 112 using, for example, a web browser or a mobile application.

[0038] In some embodiments, the object detection and location system 112 provides an image to the user system 114 that represents the location of an object detected by the object detection and location system 112. In various embodiments, the object detection and location system 112 may provide a tag or label that includes physical or semantic information about the object, such as a category or characteristic of the detected object. An example of this can be seen in Figure 20. A user may interact with an example user interface 2000 that shows an area of ​​the physical environment and an icon that indicates the space where the object is located.

[0039] In some embodiments, the model datastore 116 may be any structure or structures suitable for captured data, such as 3D models, LiDAR data, images, etc. In some examples, the model datastore 116 may be an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, etc. The model datastore 116 may store digital images or video captured by the image capture device 106. In some embodiments, the model datastore 116 may store interior and / or exterior three-dimensional models of various physical environments. The three-dimensional models may be generated by the object detection and localization system 112. In one example, the three-dimensional models may be generated by a third-party software application (not shown).

[0040] 2 shows a block diagram of an object detection and localization system 112 according to some embodiments. The object detection and localization system 112 includes a communication module 202, a communication module 204, a geolocation module 206, a depth data module 210, an alignment module 208, a mesh construction module 212, an object identification module 214, a tag module 216, a visualization module 218, an object data store 220, and an image data store 222.

[0041] The communications module 202 may send or request requests, images or data between any of the modules or data stores of the object detection and localization system 112 and devices of the environment 100 of FIG.

[0042] The communications module 204 may receive any number of two-dimensional images captured by the image capture device 106 of a physical environment (e.g., the interior of a building, facility, home, etc.). In some embodiments, the communications module 204 receives depth data of the physical environment from the image capture device 106.

[0043] In one example, two-dimensional images of the home interior (e.g., the physical environment) captured by number of image capture device(s) 106 are provided to the object detection and localization system 112. The communications module 204 may receive the two-dimensional images. In some embodiments, the communications module 204 may receive depth data of the home interior from a depth sensor (e.g., part of the image capture device(s) 106 or separate from the image capture device(s) 106).

[0044] In some embodiments, the geolocation module 206 may communicate with an external location system, such as a Global Positioning System (GPS). The geolocation module 206 may match one or more of the raw GPS coordinates (e.g., captured by the image capture device 106 or the digital device when the image(s) were captured) to a single capture location based on the time the image data was captured and the time the GPS measurements were captured at the capture location.

[0045] For example, in an implementation in which image data is captured at one or more capture locations by rotating an image capture device about a fixed axis, the two-dimensional images may be time-stamped with information indicating the start and end times of the rotation. According to this example, the geolocation module 206 may identify GPS coordinates captured between the start and end times of the rotation and associate these GPS coordinates with the capture locations.

[0046] In other implementations, the geolocation module 206 may be configured to associate raw GPS coordinates with a particular capture location based on the GPS coordinates within a defined time range relative to the image capture time at the capture location. In this regard, the geolocation module 206 may group or cluster a set of raw GPS coordinates together with a single capture location based on the associated timestamp information. Following clustering of raw GPS coordinates based on matching capture times for capture times / periods associated with each capture location, in some implementations, the geolocation module 206 may remove any GPS coordinates captured at a time that does not correspond to the image capture time (e.g., remove GPS coordinates captured between multiple capture locations).

[0047] In various embodiments, the geolocation module 206 may reside on the image capture device 106 capable of obtaining GPS coordinates or on a digital device local to the image capture device 106. In this example, the object detection and location system 112 may not include the geolocation module 206.

[0048] In some embodiments, two or more two-dimensional images received from the image capture device 106 require alignment from the alignment module 208 to generate a two-dimensional panoramic image (up to 360°) of the physical environment. In one example, the alignment module 208 aligns two-dimensional images (e.g., panoramic images) and depth data of the same physical environment to generate a three-dimensional model of the physical environment. In some embodiments, the object identification module 214 identifies three-dimensional objects from the three-dimensional model.

[0049] The depth data module 210 may determine depth data associated with two-dimensional images received by the object detection and localization system 112. In some embodiments, the depth data module 210 receives depth data from the image capture device 106 (or a depth device such as a LiDAR) in the form of a time-lapse between the detection of an outgoing light pulse and a reflected light pulse. The time-lapse determines the distance from the depth sensor to a particular point in the physical environment.

[0050] In various embodiments, the depth data module 210 resides on the image capture device 106 when capturing the depth data, or on a device local to the image capture device 106. In some embodiments, the object detection and localization system 112 may not include the depth data module 210.

[0051] The object detection and localization system 112 may use the two-dimensional images along with the depth data to generate a three-dimensional model (e.g., a 3D model) of a physical environment (e.g., the interior of a home).

[0052] In some embodiments, the depth data module 210 includes a machine learning algorithm (e.g., a neural network) that uses two or more two-dimensional images of the physical environment to estimate depth data for all or a portion of the scanned physical environment. The machine learning algorithm may use features of the two-dimensional images, such as color or light intensity, to determine the distance of pixels in the images from the image capture device. If the image capture device 106 includes multiple image sensors, such as stereo assisted imaging with multiple image sensors offset by a predetermined distance, the depth data module 210 may use the offset images of the physical environment to determine depth data for the environment captured in one or more images.

[0053] In some embodiments, before a three-dimensional model of the physical environment can be generated, a mesh of the environment is generated by mesh construction module 212. The mesh comprises a series of triangles, lines, curved surfaces (e.g., non-uniform rational B-splines ("NURBS")), quads, n-grams, or other geometric shapes that may connect a set of points. In one aspect, portions of the mesh may include image data describing texture, color, intensity, etc. In various embodiments, a captured 2D panoramic image (or a portion thereof) may be overlaid on portions of the mesh to generate a textured mesh. The textured mesh may include walls, floors, ceilings, and individual objects within a space. The textured mesh may be dimensionally accurate and allow a user of the user interface to determine the physical distance between two points in the three-dimensional model of the physical environment. In one example, mesh construction module 212 utilizes a mesh generation method such as Laplacian smoothing, parallel mesh generation, or stretched grid methods.

[0054] In one example, the mesh is a triangular mesh, and pixels within a particular triangle of the triangular mesh represent a region of the environment on a common plane. Thus, a digital 3D model of an object may be generated based on 2D image data, 2D sensory data, sensory data combined with raw 2D data, 3D spatial data (e.g., spatial depth and distance information), computer-generated position data, etc. In some embodiments, the three-dimensional model of the physical environment is generated without the use of a mesh.

[0055] In some embodiments, the object identification module 214 includes a trained machine learning (ML) algorithm to identify objects captured in one or more images of the physical environment. In one example, the object identification module 214 identifies multiple objects from various two-dimensional images of the physical environment. The objects captured by the two-dimensional images may represent instances of the physical object at one or more particular viewpoints. Each instance of the object may be associated with a three-dimensional surface in physical space that indicates a location in the 3D model.

[0056] Each 3D object detection need not be a unique object in the physical environment. It will be appreciated that the same object may be captured in different images from different viewpoints. For example, the 3D object detections 1712, 1722, and 1732 in Figures 17A, 17B, and 17C correspond to the same chair from different viewpoints, respectively.

[0057] Each distinct object identified from the multiple images may correspond to one instance of a particular object in the three-dimensional model of the physical space. Each instance of a particular object may be categorized by the object identification module 214. Various instances of the same object may be categorized as one different object. An aggregate confidence may be used to categorize a particular object using the individual categories of each of the various instances of the same object.

[0058] For example, two objects may be categorized together as "chairs" even though they are distinct from one another (e.g., one is a kitchen chair and the other is a rocking chair). A category may be referred to herein as an "object type" (e.g., both instances have the same "object type" of "chair"). In this example, both the kitchen chair and the rocking chair are distinct instances (i.e., different object instances).

[0059] In one or more embodiments, the object identification module 214 may be configured to identify objects (e.g., structural objects, fixtures, appliances, furniture, decorative objects, people, places, things, etc.) appearing in the 3D model using hardware, software, or a combination of hardware and software that automatically detects visual objects appearing in the image, video, or 3D model. For example, with reference to a 3D model showing the interior space of a house, the object identification module 214 may identify walls, floors, ceilings, windows, fixtures, appliances, furniture, etc. The object identification module 214 may further identify groups of objects that are associated with each other. For example, the object identification module 214 may identify groups of objects that are associated with a particular room (e.g., kitchen or bedroom). Object recognition algorithms rely on matching, machine learning, or pattern recognition algorithms using appearance-based or feature-based techniques. Common techniques include edge detection, gradients, Histogram of Oriented Gradients (HOG), Haar wavelets, and linear binary patterns. In various embodiments, the object identification module 214 may identify objects appearing in the image data associated with the 3D model using various models, including, but not limited to, extracted feature and boosted learning algorithms, Bag of Words (BoW) models with features such as speeded-up robust features (SURF) and maximally stable external regions (MSER), gradient-based and differential-based matching techniques, the Viola-Jones algorithm, template matching, deep learning, image segmentation, and blob analysis.

[0060] In some implementations, the object identification module 214 may determine detailed characteristics associated with the identified object to increase the accuracy of identification of tags potentially associated with each object. Detailed characteristics may include any possible characteristic of an object that may be determined by the object identification module 214. For example, detailed characteristics may relate to one or more colors of the object, the object type, the dimensions of the object, the material of the object, a manufacturer or brand associated with the object, an identity associated with the object, etc. For example, in addition to detecting an appliance contained within the 3D model kitchen, the object identification module 214 may determine that the appliance is a microwave oven and determine the microwave oven brand, the microwave oven model, and the microwave oven size.

[0061] In some embodiments, the object identification module 214 may utilize contextual information received in connection with the capture process to facilitate identification of objects contained within the 3D model. For example, in some implementations, during the capture process, a user may provide spoken notes or annotations about the environment being captured. For example, as a user scans a living room with a 3D capture device, the user may provide spoken notes about the objects the user sees and / or sees within the captured 2D / 3D image data. For example, the user may state that the living room has hardwood floors, the type of wood in the floors, the company from which the floorboards were purchased, and any other possible descriptive information the user may know that may facilitate identification of the floors and / or tag data associated with the floors when rendered in the 3D model of the living room. These spoken notes may be associated with a particular image data capture (e.g., a 2D image) and location information associated with the image data capture, including location information of pixels or points each contained within the 2D image. Additionally, the object identification module 214 may utilize sensor measurement information (e.g., temperature, pressure, light, etc.) associated with the captured 2D or 3D image data of the environment to facilitate identification of objects contained within the 2D or 3D image data. Thus, in some embodiments, the object identification module 214 may utilize contextual clues previously associated with the image data of the 3D model to identify and characterize objects within the 3D model.

[0062] Additionally, rather than attempting to recognize and characterize all possible taggable objects contained within the 3D model, the object identification module 214 may utilize contextual annotation information received during the capture process that indicates which objects should be specifically identified and characterized for tagging purposes. For example, this contextual annotation information may include verbal annotations provided by the user during the capture process, as well as image-based contextual information. For example, the user may take a close-up photograph or video of the object and / or a logo on the object to facilitate automated object recognition of the object and indicate that a tag is desirable for association with the object. In other embodiments, the user may provide input to select a point or set of points contained within the 2D or 3D image data associated with the 3D model (including, for example, image data captured prior to alignment and representation of the 3D model generated after generation of the 3D model) to which the user wishes to apply a tag. The object identification module 214 may be further configured to analyze visual and structural features associated with the point or set of points to determine and characterize the associated object.

[0063] In various embodiments, the tag module 216 may define tags for one or more objects and associate the tags with specific points in the three-dimensional model. In one or more implementations, the tag module 216 may label or tag specific points, regions, objects of interest, object instances, etc. The tags may be shown in a visualization by the visualization module 218 (discussed herein). The tags may include or be text, images, video, audio, and / or hyperlinks. In various embodiments, the tag module 216 identifies objects, paths, or points of interest in the visualization of the 3D model (e.g., by tags and / or labels). The tag module 216 may identify tagged objects from the object identification module 214.

[0064] In some embodiments, a user interacting with the GUI and / or visualization of the 3D model may associate tags with objects, points, etc. in the 3D model. For example, after selecting or identifying a point, region, or object of interest, the user may generate, upload, attach, embed, link, select, or otherwise provide input to tag module 216 that includes a tag for association with the point, region, or object of interest. In some embodiments, the tag is used to describe or provide further information (e.g., semantic information, emotional information, historical information, ancillary information, etc.) about the point, region, or object with which the tag is associated.

[0065] In some embodiments, the tag module 216 may associate further metadata with the tag, such as information about the creator or creator of the tag, a timestamp associated with the generation of the tag, a three-dimensional model associated with the tag, the location of the tag relative to the 3D model, the location of the tag relative to the physical environment (e.g., floor), object type or object category, etc. In general, a tag may be applied to or associated with a point, region, or object within a 3D model. For example, in some implementations, a tag may be associated with an individual 3D point associated with a 3D model. A tag may also be associated with a group of 2D or 3D points on or within a 3D model. In another example, a tag may be associated with a face / surface or set of faces / surfaces contained within a 3D model. In another example, a tag may be associated with an edge of a group of edges located on a structural surface of a 3D model. In another example, a tag may be associated with a volume or group of volumes enclosed within or including one or more structural surfaces in a 3D model. A tag may be associated with a region or volume of free space within a model. A tag may be applied to or associated with a visual image or set of images that are components of a 3D model. In another aspect, a tag may be associated with a particular viewpoint for a 3D model. For example, when applied to an architectural building, a tag may be associated with a particular viewpoint of the building, a room within the building, a wall within the building, a fixture within the building, a piece of furniture within the building, a decoration within the building, etc. The size, shape, and appearance of the tag icon may vary. In some embodiments, the size, shape, and appearance of the tag icon for a tag may be tailored based on the preferences of the author or the system applying the tag. For example, an author may select a particular symbol to represent the tag, the size of the symbol, the dimensions of the symbol, the color of the symbol, the opacity or tilt of the symbol, etc. In other implementations, different types of tag icons and / or appearances of the tag icon may be employed to distinguish between features of the tag represented by the tag icon. For example, different types or colors of the tag icon may be employed to identify different authors or entities that have applied the tag. In another example, different types or colors of tag icons may be employed to indicate the type of tag data or tag metadata, such as whether the tag provides text data, image data, audio data, video data, hyperlink data, etc.For example, musical symbols may be employed to represent tag data including music or photographs on a film tape, or a video camera may be employed to represent tag data including video. In another example, different types or colors of tag icons may be employed to designate a class or category associated with the tag. In another example, different types or colors of tag icons may be employed to indicate a timestamp associated with the tag.

[0066] The particular tags and associated tag icons included within a representation of a 3D model presented to a user may vary based on a variety of factors, including factors related to the preferences of the creator or entity that applied the tags, as well as factors related to the user viewing / consuming the 3D model and the viewing context. For example, different points, regions, or objects within a model may be associated with different tags. Additionally, a single point, region, or object within a 3D model may be associated with multiple tags. The particular set or subset of tags and associated tag icons associated with a 3D model presented to an end user may vary based on the characteristics of the end user (e.g., preferences, demographics, etc.), the client device on which the 3D model is presented, and the viewing context (e.g., location, time of day, day of the month, mobility status, etc.). In some embodiments, the entity that applied the tags, or an entity with authority over how the tags and associated tag icons are rendered to an end user, may specify when one or more of the tags and associated tag icons are presented to an end user. In response, the tag module 216 may integrate and / or display tags and associated tag icons in the representation of the 3D model. For example, the entity may customize the particular tags and / or tag icons presented to an end user based on the characteristics of the end user (e.g., preferences, profile, demographic characteristics, etc.), the characteristics of the client device (e.g., display type and size), the viewing context (including, e.g., location, time of day, season, mobility status, etc.), and the like.

[0067] In some embodiments, interaction with a particular tag icon may trigger a response that is associated with one or more related tags. For example, in some implementations, tags may be grouped based on various factors. For example, tags may be grouped based on association with the same room, the same type of object, the same date, the same manufacturer, the same price factor, the same creator, the same decorator. According to these embodiments, selection or interaction with one of the tag icons for a tag included in a defined group may cause any number of other tags in the group to assume a particular state, such as activation, highlighting, hiding, changing size, shape, or position, etc. For example, in one implementation, selecting one tag icon for a tag included in a tag group may display the tag icon only for that tag in the group. Other tag icons in the model that are not associated with the group may be removed from display or displayed in a lighter or more transparent color. In another example, selecting one tag icon for a tag in a tag group may highlight the tag icon only for that tag in the group. In another example, selecting a tag icon for a tag in a tag group may display tags for all tags in the group in a side panel. In one example, a user may submit a search query for outlets in a three-dimensional model of the house 110 in the search field 402 of FIG. 4. The object detection and locating system 112 may extract the search query. A request to search the object data store 220 for an object type input and an object instance input corresponds to the search query. In the example of FIG. 4, the object data store 220 may receive a search query for an outlet. A search of the object data store 220 is performed for object instance inputs that are categorized or identified as an outlet. In some embodiments, each object instance input is associated with one particular outlet found in the house 110. Each object instance input may include identifiable information about the object, such as the object's category and a unique object identifier and characteristics. Additionally, the object data store 220 may perform a search for object type inputs that correspond to the outlet. The object type inputs may include an incremental total number of outlets in the 3D model of the physical environment.

[0068] The visualization module 218 may enable visualization or walk-through of a 3D model. The 3D model may be generated based on two-dimensional images provided by one or more image capture device(s) 106 and / or depth data. In various embodiments, the visualization module 218 provides (e.g., via download or streaming code) the visualization or walk-through to one or more other digital devices with which the user interacts (e.g., walk-through 3D model).

[0069] In some embodiments, the visualization module 218 provides interaction through visualization, by API, or other means. In one example, a user may provide a query or interact with GUI elements provided by the visualization module 218 to request information about one or more objects identified by the object identification module 214. The request may be for several different object types (e.g., number of appliance objects on a floor or in a home), the location of one or more different object instances in a 3D model or physical environment, etc. The visualization module 218 may extract information already stored by the object identification module 214 and / or the tag module 216 and provide a response. In some embodiments, the response may be in the form of text (e.g., a list or number) and / or may use a visualization to indicate the location of the object instance(s) that respond to the request.

[0070] The object data store 220 may be any one or more configurations suitable for storing object instance entries or object type entries. For example, the object data store 220 may be an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, etc. The object data store 220 stores information regarding location, object type, or object count identified by the object identification module 214.

[0071] In some embodiments, the object data store 220 stores information about objects in an object instance entry. An object instance entry may include or indicate a particular instance of an object type. An object type entry may be associated with a particular object type. In some embodiments, the object data store 220 may store an object type count indicating a total number of objects of that object type on a floor of the 3D model, a portion of the 3D model, and / or the entire 3D model. For example, if the object identification module 214 identifies an instance of a dining room chair, the object identification module 214 may send a request to the object data store 220 to generate an object instance entry associated with this particular dining room chair. Additionally, the object identification module 214 may augment an increment total number of object type entries associated with dining room chairs.

[0072] Each object instance entry is associated with one object identified by the object identification module 214. The object instance entry may include identifiable information about the object, such as a category and a unique object identifier. The object instance entry may also include physical characteristics of the object, such as 3D coordinates (a point in 3D Euclidean space), the object's estimated size and color, and the object's brand (if applicable). In some embodiments, the object data store 220 includes characteristics of a tag or icon representing the object in the three-dimensional model, such as the size, shape, or location of the icon representing the object.

[0073] The image data store 222 may be in any one or more configurations suitable for digital images, including 2D panoramic images, 3D panoramic images, 3D models, etc. Like the object data store 220, the image data store 222 may include or be an active database, a relational database, a self-referential database, a table, a matrix, an array, a flat file, a document oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, etc. In some embodiments, the object data store 220 stores depth data generated by the depth data module 210. In various embodiments, the object data store 220 stores characteristics associated with an image capture device or characteristics associated with each of multiple image or depth captures used to determine the 3D panoramic image.

[0074] 3 shows a flowchart of an object detection process 300 according to some embodiments. In various embodiments, the communications module 204 receives multiple two-dimensional images and / or three-dimensional models of the physical environment. In some embodiments, the communications module 204 receives any number of two-dimensional images from the image capture device 106. The communications module 204, in some embodiments, may receive depth information from the image capture device 106 or other devices.

[0075] If received, the 3D model may be generated by a third party software application (not shown). In some embodiments, the communications module 204 receives the two-dimensional images and / or three-dimensional models from other sources, such as the model data store 116. In one example, views 700, 800, 900, and 1000 in Figures 7-10 each show an example of a collection of two-dimensional images of a physical environment from various viewpoints.

[0076] In some embodiments, the alignment module 208 may align or "stitch together" respective 2D images captured by a 2D / 3D panoramic capture device (e.g., image capture device 106) to generate multiple two-dimensional images. For example, the alignment module 208 may also utilize GPS coordinates of 106 during capture of each two-dimensional image to determine alignment of one or more two-dimensional images.

[0077] The alignment module 208 may also employ depth information regarding the relative positions of each visual feature contained within each 2D image to further align each 2D image with one another in relation to the 2D panoramic image. The alignment module 208 may further perform blending and / or graph cuts at edges to remove seams. In various embodiments, the alignment module 208 may utilize features or characteristics of the two-dimensional images, such as color or light intensity, to align or stitch the images together.

[0078] In some embodiments, the object detection and localization system 112 may generate a 3D model based on any number of 2D images and / or depth data.

[0079] In step 302, the object identification module 214 may identify an object. In various embodiments, the object identification module 214 identifies an object based on the 2D image of the image capture device and associated position information, depth data, and / or mesh data. These various steps are discussed herein. The object identification module 214 may identify an object captured in the 2D image and / or the 3D model.

[0080] In step 304, the tag module 216 may categorize and / or tag objects in the 3D model as object instances and / or by object type. In various embodiments, the object identification module 214 and / or the tag module 216 collect information regarding the location of the object(s) identified in the 3D model by the object identification module 214, the number of object instances for one or more object types, etc.

[0081] In step 306, information collected or generated by the object identification module 214 and / or tag module 216 may be stored in the object data store 220 so that the information can be later extracted and provided in response to an information request (e.g., from a user).

[0082] In step 308, the visualization module 218 provides the 3D model visualization to a digital device or allows a user to view and / or interact with the 3D model (e.g., as a walkthrough of a virtual environment). The visualization may be interactive, where the visualization includes a graphical user interface (GUI) and allows a user to request information (e.g., through a query field) or interactive elements of the GUI.

[0083] In step 310, the visualization module 218 receives an information request from a user (e.g., or another digital device). The information request may be for the number or location of a particular object type within the 3D model.

[0084] In step 312, the visualization module 218 extracts information (eg, from the object data store 220 and / or the image data store 222) and responds to the request.

[0085] In step 314, the visualization module 218 responds to the request by providing a response and / or a visualization of the object in the 3D model.

[0086] Figure 4 is an example user interface 400 associated with a visualization. In Figure 4, a user can interact with an area 410 to view and move around portions of a building. In this example, the user requests more information about the location of the outlets in the 3D model by providing a query in a query field of the visualization (e.g., provided by the visualization module 218).

[0087] In response, the visualization module 218 may extract information about objects of the requested object type ("outlets") and provide a list 420 identifying the outlets and / or a view that includes the outlets in an image (e.g., a dollhouse view).

[0088] Although an electrical outlet is used in the example of FIG. 4, it will be appreciated that a user may explore any object and associated information identified by the object detection and location system 112.

[0089] Providing information about the 3D model and associated physical structure may enable a user to do more than "walk through" the 3D model. Rather, the user may interact with the visualization and identify key aspects of the subject model (e.g., number of bathrooms, location of sinks, etc.) to ensure that the property is suitable for the user's desired purposes.

[0090] By enabling automated object detection and object information collection, the object detection and location system 112 may provide users with interactive visualization and analysis capabilities not previously available in a manner that tailors any number of 3D models for any number of users or other digital devices. Additionally, identifying objects and collecting information as described herein reduces errors, makes information more readily available, and conserves computer resources significantly over manual data collection methods.

[0091] In some embodiments, a user may interact with list 420 that results in multiple object instances of a given object type (ie, category of objects).

[0092] Returning to the example of Figure 4, list 420 lists identified outlets located within the 3D model. A user may interact with one particular outlet, such as entrance 422. In response, user interface 400 brings up the location of the particular outlet in the building floor plan.

[0093] 5 illustrates a flowchart of an object detection process 500 according to some embodiments. As described herein, in various embodiments, the communications module 204 receives a number of two-dimensional images and / or a three-dimensional model of the physical environment. In some embodiments, the communications module 204 receives any number of two-dimensional images from the image capture device 106. The communications module 204 may, in some embodiments, receive depth information from the image capture device 106 or other devices. In some embodiments, the object detection and localization system 112 may generate a 3D model based on any number of 2D images and / or depth data.

[0094] In step 502, the object identification module 214 may identify an object. In various embodiments, the object identification module 214 identifies an object based on the 2D image of the image capture device and associated position information, depth data, and / or mesh data. These various steps are discussed herein. The object identification module 214 may identify an object captured in the 2D image and / or the 3D model.

[0095] In step 504, the tag module 216 may categorize and / or tag objects in the 3D model as object instances and / or by object type. In various embodiments, the object identification module 214 and / or the tag module 216 collect information regarding the location of the object(s) identified in the 3D model by the object identification module 214, the number of object instances for one or more object types, etc.

[0096] At step 506, the object identification module 214 may increment an object type total count (e.g., number of object types) for each object instance of a particular object type. For example, the object identification module 214 may increment an outlet total count (e.g., an object type of "outlet") for each outlet instance identified by the object identification module 214 with a particular 3D model or physical environment.

[0097] In step 508, the object identification module 214 may collect information regarding the locations of any number of object instances identified by the object instance module 214. For example, the object identification module 214 may identify the location of an object based on the location of an image capture device that captured an image showing the object, depth data, and / or a mesh of the 3D model. The location information may indicate a floor or portion of the 3D model in which the object instance is located. In some embodiments, the location information indicates a location for a particular object instance within the 3D model.

[0098] The object identification module 214 may associate count and / or location information of object types with object instances and store the information in the depth data module 210 to enable the visualization module 218 to respond to queries about the object(s) in the 3D model.

[0099] In step 510, the visualization module 218 provides the 3D model visualization to a digital device or allows a user to view and / or interact with the 3D model (e.g., as a walkthrough of a virtual environment). The visualization may be interactive, whereby the visualization includes a graphical user interface (GUI) that allows a user to request information (e.g., through a query field) or interactive elements of the GUI.

[0100] In step 512, the visualization module 218 receives a query from a user (e.g., or other digital device) over the API or visualization field. In this example, the user may provide a query regarding the number and location of a particular object type (e.g., fire extinguishers).

[0101] In step 514, the visualization module 218 extracts information (e.g., from the object data store 220 and / or image data store 222) and responds to the request. In this example, the visualization module 218 identifies the object type of the request (i.e., fire extinguishers) and extracts the total number of object types of the 3D model (e.g., number of fire extinguishers). It will be appreciated that the user may request the number of fire extinguishers per floor, per location of the 3D model, etc. In this example, the visualization module 218 may limit or filter the information request to extract only the number of object instances that qualify for the query. In one example, the visualization module 218 may identify qualifying object instances for the object type identified in the query that are located at or within a particular location of the model, determine the total number of qualifying object instances, and return the number of qualifying object instances.

[0102] The query may further request the location of one or more fire extinguishers within all or a portion of the 3D model. The visualization module 218 may extract object information indicating the location of the target object (i.e., the location of the object instance associated with the object type of “fire extinguisher”).

[0103] In step 516, the visualization module 218 may provide the extracted information (e.g., the number of object instances of the desired object type) and the locations of the satisfying object instances. In some embodiments, the visualization module 218 may provide a list of object locations and / or a representation of where the object is located within the 3D model (e.g., in a dollhouse view, the user may have a perspective view of a portion of the 3D model, one or more floors of the 3D model, or the entire 3D model).

[0104] 6 is a flow chart for identifying the same or different object instances in two-dimensional images based on the position of the image capture device that captured the images in some embodiments. While there may be many ways to determine whether objects in multiple images are the same object (e.g., using only depth data and the position of the depth data in a 3D model), the process described in FIG. 6 is scalable as the number of images and objects in the images increases.

[0105] When the same object is identified in multiple images, the representation of the object in the various images (e.g., various views) may be used in the 3D model. For example, a user may control the viewpoint in a walk-through of the 3D model. As the viewpoint changes, the view of the same object (e.g., the same chair) may change depending on the viewpoint. An image of a chair that includes various representations in the 3D model may be generated based in part on the representation of the various viewpoints of the object in the various images. By using the camera position and orientation to identify the same object in the various images, pixels and other information may be presented in the 3D model.

[0106] Additionally, multiple objects may be distinguished from one another by associating them with the position and orientation of the image capture device. For example, two objects captured in different images of the same physical environment may be similar to one another (i.e., the two objects may be of the same object type). However, based on the position of the image capture device, it may be determined that the two objects are different from one another (e.g., the objects shown are both chairs but on different floors), thereby allowing similar objects to be distinguished, increasing the total number of this object type, and allowing metadata associated with the different objects (e.g., location relative to the 3D model and / or physical environment) to be stored.

[0107] At step 602, the communications module 204 receives a plurality of two-dimensional images of a physical environment (e.g., the interior of a facility, building, or residence), each of which may be associated with a position and / or orientation of the image capture device that captured the image.

[0108] In step 604, the object identification module 214 may recognize and / or identify a first object shown in a particular image. The first object shown may be from a particular view. The view of the object shown in a particular image may be at a particular angle relative to the first object, may be partially obscured in the image, etc. The first image may be associated with a position and / or orientation of the image capture device 106 that captured the image (i.e., when the image was captured by the image capture device 106).

[0109] In step 606, the object identification module 214 may recognize and / or identify a second object shown in a particular image. The second object shown may be from a particular view (e.g., a different view than that shown in the first image). The view of the second object shown in a particular image may be at a particular angle relative to the second object, may be partially obscured in the image, etc. The second image may be associated with the position and / or orientation of the image capture device 106 that captured the image (i.e., when the second image was captured by the image capture device 106). The position of the image capture device 106 may be different from the position of the image capture device 106 when the first image was captured.

[0110] The location of the first object and the second object may be determined in any number of ways. In some embodiments, the location of the first object and the location of the second object may be determined, at least in part, using depth data (if available) from a depth data sensor (e.g., LiDAR). In various embodiments, the image capture device 106 includes both a camera and a LiDAR and is configured to take images and depth data (e.g., measurements using the LiDAR sensor) of the physical environment. Objects shown in the images may be associated with the depth data to identify the location, and / or the location may be identified if two or more objects in two or more images show the same object (e.g., by determining that the objects in each of the multiple images match the same location defined by the depth data).

[0111] It will be appreciated that many objects may look similar but are different instances of a similar object (e.g., there are two kitchen chairs that look fairly similar, or there are two electrical outlets that appear to be different designs based on the images). Similarly, some objects may look the same (e.g., there are fire extinguishers on every floor that are the same model extinguisher) but are different (can be distinguished based on location).

[0112] It will be appreciated that any number of images may be taken by any number of image capture device(s) 106 (e.g., multiple images may be captured by repositioning the same image capture device 106, or any number of image capture devices 106 may be used to capture any number of images from similar or different positions in the physical environment).

[0113] The object identification module 214 may identify objects in an image in any number of ways. For example, the object identification module 214 may utilize a trained neural network (e.g., R-Mask CNN) to identify any number of objects (e.g., using instance segmentation). In some embodiments, the neural network generates a confidence score that is associated with the confidence with which it identified and / or categorized the object. In various embodiments, the object identification module 214 may apply a confidence threshold and remove the object identification(s) and / or classification if the confidence score is insufficient relative to the confidence threshold.

[0114] In step 608, the object identification module 214 maps a pixel (e.g., a portion of the image associated with the first object) to a location within the 3D model and / or physical environment based at least in part on the position and / or orientation of the image capture device 106 when the first image was captured.

[0115] At step 610, the object identification module 214 determines whether the second object in the second image is a second view of the same object in the first image (e.g., the second object is a different view of the first object). In some embodiments, the object identification module 214 maps a pixel (e.g., a portion of the image associated with the second object) to a location in the 3D model and / or physical environment based at least in part on the position and / or orientation of the image capture device 106 when the second image was captured. If the mapping matches, the first object and the second object are of the same object type (e.g., categorize them as part of the same object type) and the object identification module 214 may conclude that the first object and the second object in the different images represent the same object in the physical environment.

[0116] In various embodiments, rather than mapping the pixels or images of the second image, the object identification module 214 may use the position and / or orientation of the image capture device 106 at the time the second image was captured to determine whether the first object and the second object match within the 3D model and / or physical environment.

[0117] If the object identification module 214 determines that the objects are the same, then in step 612 the 3D model may utilize pixels and / or portions of the images of the various views of the same object within the 3D model (e.g., within the visualization of the 3D model) to enhance the accuracy and realism of the object shown in the 3D model. In various embodiments, the object identification module 214 may collect object information about a single object shown in both images. For example, the object identification module 214 may store metadata associated with the object, such as object type, augment a count of a particular object type and / or object location. The object information (e.g., associated with the object) may be stored in the object data store 220.

[0118] If the object identification module 214 determines that the first object and the second object are different (e.g., the first object and the second object are in different locations in the physical environment), then in step 614 the object identification module 214 may collect individual object information for each object. In various embodiments, the object identification module 214 may collect object information for both objects (e.g., the first object and the second object) shown in the different images. For example, the object identification module 214 may store metadata associated with both objects, such as object type, augment the number of each object of a particular object type, and / or the location of the objects. The object information (e.g., associated with each object) may be stored in the object data store 220.

[0119] In various embodiments, pixels in different views of the same object shown in different images may aid in visualization of the object in the 3D model. Additionally, identifying objects in images of the physical space preserves metadata (e.g., tagging, location, etc.) associated with different objects, enabling querying of the 3D model and / or the physical space.

[0120] Information associated with various objects (e.g., location, number of the same type of object, etc.) may be collected and stored. This enables an interactive system where a user may query for some specific objects (e.g., "How many electrical outlets are there?") and / or the location of specific objects (e.g., "Where are the fire extinguishers located?") in the 3D model. The system may extract answers from the collected information associated with the locations and numbers / object types of different objects and provide textual and / or visual answers.

[0121] Figures 7-10 show various images taken at various locations in a physical environment in one example. In the examples of Figures 7-10, the images were taken at a warehouse or retail store. Figure 7 shows a view 700 that includes a region that also appears at various angles and perspectives from other images, for example, Figures 8-10. In Figure 7, an object 702 is partially visible.

[0122] FIG. 8 shows a view 800 in the same physical environment as that shown in FIG. 7, which may or may not include the same objects.

[0123] Figure 9 shows a view 900 in the same physical environment as shown in Figure 7, but including an object 902. Object 902 may be object 702. In various embodiments, object identification module 214 may use object recognition (e.g., using instance segmentation) to recognize objects in both images and may utilize orientation and / or position information of the image capture device 106 that captured the images to determine whether the same object appears in both images.

[0124] FIG. 10 illustrates a view 1000 in the same physical environment as the physical environment shown in FIGS. 7-9. The object 1002 identified in FIG. 10 may be object 702 and / or object 902. In one example, the object identification module 214 may determine whether the objects are the same object type, and if they are the same object type, the object identification module 214 may determine whether the objects are the same based on the position and orientation of the capturing image capture device(s) 106. In various embodiments, the object identification module 214 may recognize the objects in both images using object recognition (e.g., using instance segmentation) and may utilize the orientation and / or position information of the image capture device 106 that captured the images to determine whether the same object appears in any number of images.

[0125] Figure 11 illustrates a merge of the image collections of Figures 7-10, showing the location of the image capture device and the location of the detected two-dimensional object. In this example, different capture locations 1110, 1112, 1114, etc. may be locations where images were taken that include the object 1102. Each capture location 1110, 1112, and 1114 may be the location of the image capture device 106 that captured the image. In various embodiments, the orientation of the image capture device 106 when the image was captured is also associated with the image.

[0126] For example, object 1102 may be the same object as objects 702, 902, and 1002. In this example, each of the various capture locations may correspond to a capture location of one of the images of Figures 7, 9, and 10. Based on the position and orientation of the object image capture device 106 when the images were captured, the objects may be triangulated to confirm or determine (e.g., with a reasonable degree of certainty) that similar objects in the various images may be the same object (or, alternatively, may be different objects).

[0127] 12 shows a flowchart for collecting information and providing interaction with a 3D model in some embodiments. In step 1202, the object identification module 214 categorizes or recognizes objects in one or more images. The object identification module 214 may categorize the objects and / or store object information (e.g., the location of the object in the 3D model or the physical environment) in the image data store 222.

[0128] In some embodiments, the object identification module 214 may utilize a neural network to identify objects in a two-dimensional image (e.g., of the physical environment captured by the image capture device 106). It will be appreciated that one or more two-dimensional images may contain one or more identified objects.

[0129] As described herein, the same object to be identified may be identified in different two-dimensional images from different viewpoints. In some embodiments, the object identification module 214 may generate a bounding box that indicates the boundary or outer extent of a particular region of interest (which may, for example, contain the object to be identified or recognized).

[0130] FIG. 14 shows an overview of process 1400 of one implementation of an object detection and localization system. Element 1402 is a 3D model, and can be both an output and an input to process 1400. Element 1404 of process 1400 can perform 2D processing including Neural Network 1, Neural Network 2, and Neural Network 3 (which can be differently trained neural networks) to train different neural networks to recognize different objects. Element 1406 can associate pixels of the identified object with the three-dimensional surface of the 3D model. Element 1408 can merge multiple detections, and can perform object type categorization based on the detections and semantic information. Process 1400 can tag one or more recognized objects with tags located within the visualization of the 3D model.

[0131] In some embodiments, the neural network may perform object recognition (e.g., using instance segmentation) and provide the probability that a particular object in an image belongs to a particular category (e.g., object type).

[0132] For each object recognized by the neural network, the object identification module 214 may identify pixels in each of the subsets of 2D object detections that are associated with the particular object. These identified pixels may be known as object pixels. In addition to identifying object pixels for one or more objects, the object identification module 214 may associate the object pixels with surfels or surface elements of a 3D model of the physical environment.

[0133] For each object, the object identification module 214 may obtain collected object data (e.g., the location and / or orientation of the image capture device 106 that captured the image of the object, an extrapolation of the object position relative to one or more locations and / or orientations, etc.). Determining the proximity of a subset of two-dimensional object detections to one another may indicate a clearer probability that two or more objects recognized in different images are the same object.

[0134] In some embodiments, the object identification module 214 may provide a cumulative confidence of a particular object identified in a collection of 3D object detections based on the 3D object probability. The cumulative confidence may be an aggregate confidence, an average confidence, a weighted average confidence, etc. In various embodiments, the aggregate confidence is calculated according to the formula:

[0135] TIFF2025507782000002.tif16170

[0136] where c i is the confidence reported by the i-th object detection in the ensemble of 3D object detections for a particular object.

[0137] 17A-17C show the same chair in different images, and the probability scores that the object is recognized. In the example of FIG. 17A-17C providing ensembles 1712, 1722 and 1732 of 3D object detections, with 3D object probabilities of 0.86, 0.62 and 0.94, respectively. The aggregate confidence is: Reliability 結合 =1-(1-0.86)(1-0.64)(1-0.94)=0.997 It can be determined by:

[0138] Once the object identification module 214 has determined the cumulative confidence of a particular object, the object identification module 214 may send a request to the object data store 220 to update or create an object instance entry for the particular object (e.g., that the objects are all the same object instance, that the objects are of the "chair" object type, and / or that the object instance is associated with a location within the 3D model and / or the physical environment). Additionally, the object identification module 214 may send a request to the tag module 216 to define a tag and a particular point within the three-dimensional model to associate the tag.

[0139] 20 shows a user interface 2000 including tags 2010, which provide categories for a particular object, and tag icons 2020. A user may interact with the user interface 2000 in the area of ​​1720 to receive more information about this particular object. It will be appreciated that the sweep and confidence values ​​may or may not be provided to the user in the user interface 2000.

[0140] In some embodiments, three-dimensional detections of the same object from two different views may be represented in a 3D model (e.g., a detection surface that can be described in the same coordinate system given prior knowledge of the relative positions of the image capture devices that viewed and captured the images) as two different sets of points (e.g., surfels). Thus, the object detection and localization system 112 may "merge" the detection surfaces from the various views (i.e., associate the union of the two sets of points with the same physical object) and represent them in a visualization of the 3D model.

[0141] One example of a technique for achieving this may assume that a surface of an object seen from one view (e.g., the surface of a table) may be close to or overlap another surface of the same object seen from a different view (e.g., the underside of the same table). The object identification module 214 may automatically determine that the detection of these two surfaces representing the same object may leverage such an assumption to arrive at a decision of the object identification module 214. The object identification module 214 may consider, for example, the distance between the centers of the two detection surface points and that the two detection surfaces are close to each other, or perhaps that each point in one of these detection surfaces has a sufficiently large number of points from the other detection surface that are located sufficiently close to the detection surface.

[0142] FIG. 14 shows an overview 1400 of one implementation of an object detection and localization system. Element 1402 corresponds to step 302 of object detection process 300. Element 1404 of overview 1400 includes Neural Network 1, Neural Network 2, and Neural Network 3 of overview 1400, where multiple 2D detections corresponding to each neural network correspond to step 306, where different neural networks are trained to recognize different objects. Element 1406, where a pixel is associated with each 2D object detection with a 3D surface of the model, may correspond to step 310. Element 1408 may correspond to step 312, where a set of 3D object detections is identified. Returning to step 1204 of FIG. 12, if there are multiple different objects of the same object type (e.g., recognized as the same category or the same object type) (e.g., located at different positions relative to the physical environment and / or the 3D model), the number of object types may be incremented for each different object of this object type to track the amount of object types. There can be counts of different object types of the same object type. For example, there can be a total object type count, to track the total number of objects of the same object type in the 3D model. There can be object type counts for portions of the 3D model (for example, to track the total number of different objects of the same type in a room, on a floor, etc.).

[0143] In step 1208, the visualization module 218 may provide an interface that allows a user to interact with the 3D model. The visualization module 218 may receive a query from the user through the interface or through a different means (e.g., through an API). A query may request information about one or more objects. For example, a query may request several specific object types in the 3D model, the location(s) of objects of the same object type, etc.

[0144] In step 1210, the visualization module 218 may extract numbers of object types from the object data store 220 for the desired object types identified in the query and location information (e.g., the number of outlets and the locations of the outlets on the floor of the physical environment) identified in the query and associated with the requested object types.

[0145] In step 1212, the visualization module 218 provides a response to the user. For example, the visualization module 218 may provide the number of object types (e.g., number of outlets) and the locations and / or positions of object instances of the same object type identified in the query. In various embodiments, the visualization module 218 may provide the information by text (e.g., a list), a display in a 3D model, graphics, audio, animation, etc.

[0146] 13 shows a flow chart of an object detection process 1300 for an object detection process according to some embodiments. In step 1302, the communication module 204 receives a previously generated three-dimensional model of the physical environment. The three-dimensional model may be generated by a third-party software application (not shown). The three-dimensional model may be generated based on two-dimensional images and depth data of the physical environment. In some embodiments, the three-dimensional model does not include a mesh.

[0147] At step 1304, the mesh construction module 212 may generate a mesh of the physical environment. The mesh construction module 212 may generate the mesh based on the received depth data of the physical environment. In some embodiments, the mesh construction module 212 generates a triangular mesh, an example of which may be found in FIG. 15A. FIG. 15A shows an image of the physical environment with a triangular mesh overlaid. The pixels within mesh elements 1502 in FIG. 15A represent regions of the environment on a common physical plane.

[0148] In step 1306, the object identification module 214 may identify objects in the three-dimensional model received in step 1302. In some embodiments, a machine learning algorithm is trained to identify three-dimensional and / or two-dimensional objects. The object identification module 214 using a trained machine learning algorithm (e.g., neural networks and / or any other ML technique) may detect and identify three-dimensional objects based on the 3D model and meshes received and / or generated in step 1304.

[0149] The object identification module 214 may identify objects within the mesh (e.g., in step 1306) and categorize the objects using a machine learning algorithm (e.g., the neural network(s) discussed herein) in step 1308.

[0150] As described herein, meshes can aid in defining locations and distances within a 3D model. For example, view 1500 in FIG. 15A shows one view of a three-dimensional model that includes mesh elements 1502. The object identification module 214 can identify two pieces of furniture: sofas 1504 and 1506. Each of the sofas 1504 can be shown on view 1500 as a series of triangles that include mesh elements 1502 shaded in different colors.

[0151] The object identification module 214 identifies pixels of the three-dimensional model that are associated with the detected three-dimensional object. When the object identification module 214 determines that a three-dimensional view in the collection of previously identified objects is of the same object, the object identification module 214 may refine an edge of the identified object. For example, the object identification module 214 may define a mesh surface of a sofa 1504 that includes mesh element 1508. The three-dimensional surface in physical space that corresponds to mesh element 1508 is not part of sofa 1504. The edge or boundary of sofa 1504 may be further refined as the object identification module 214 views sofa 1504 from various views of the three-dimensional model.

[0152] The view 1520 of FIG. 15B includes a more refined representation 1510 of the edge of the identified couch 1504 of FIG. 15A, and does not include the mesh element 1508 shown in FIG. 15A.

[0153] Views 1600 and 1620 of Figures 16A and 16B, respectively, provide another example of an object identified by the object detection and localization system 112. The object identification module 214 may identify a shape representing the identified object, namely a bed 1602, including mesh elements 1604. In this example, the representation 1606 of the bed 1602 identified in Figure 16A covers more of the actual physical space occupied by the particular piece of furniture. As shown in Figure 16B, an area 1608 of the representation 1606 is not part of the bed, yet the object identification module 214 identified this area as part of the same identified object. With fewer views of the three-dimensional model of the bedroom, the edges or boundaries of the identified object will not be aligned with reality. In some embodiments, the edges of the identified object may be manually modified by a user of the object detection and localization system 112.

[0154] The object identification module 214 may provide a cumulative confidence of a particular object being identified in a collection of 3D object detections based on the 3D object probability.

[0155] In step 1310, the object identification module 214 may send a request to the object data store 220 to update or create an entry for an object instance for a particular object. Additionally, the object identification module 214 may send a request to the tag module 216 to define a tag and a particular point in the three-dimensional model to associate the tag.

[0156] 18 shows a flowchart of an object detection process 1800, according to some embodiments. In optional step 1802, the image capture device 106B may capture a digital image of a building, such as the house 110. In step 1804, the communication module 204 may receive a captured two-dimensional image of the house 110. An example of a two-dimensional image captured by the image capture device 106B may be the two-dimensional panoramic image 1900 of FIG.

[0157] In this process, the communication module 204 receives a two-dimensional image of the interior and / or exterior physical environment of the house 110. Prior to identifying an object shown in the digital image of the house 110, the object detection and localization system 112 may estimate depth data associated with the two-dimensional image. In step 1806, the depth data module 210 may determine depth data associated with the two-dimensional image received by the communication module 204. The depth data module 210 may include a machine learning algorithm that estimates the depth data of the two-dimensional image based on features such as color or light intensity and determines the distance between a spatial point shown in a particular pixel of the image and the image capture device that captured the image. The estimated depth data may be used to generate a three-dimensional model of the physical environment. An example of a depth map of the physical environment can be seen in FIG. 19. The depth map 1910 maps each pixel and measures the distance from the optical center of the image capture device to the nearest physical surface that intersects with a ray traced from the camera's focus through a given pixel.

[0158] In step 1808, the object identification module 214 utilizes an artificial neural network to identify objects in the two-dimensional panoramic images. In some embodiments, the artificial neural network identifies objects in the two-dimensional images. A two-dimensional image may contain one or more identified objects. The same identified object may be identified in various two-dimensional images from various viewpoints. In some embodiments, the object identification module 214 may generate a bounding polygon that indicates the boundary or outer extent of the identified object. An example of a bounding polygon is the square that indicates the boundary of the chair in the three-dimensional object detection 1722 of FIG. 17B. In some embodiments, the object identification module 214 identifies a two-dimensional image subset of the received two-dimensional images for each identified object. Each image of the two-dimensional image subset includes pixels that are indicative of a particular object found in the physical environment. A particular object may be found in more than one two-dimensional image. Each two-dimensional image may contain more than one object. Each two-dimensional object detection represents an instance of a physical object as seen by the image capture device. An example of 2D object detection is 3D object detection 1712 in FIG. 17A, which shows the bounding box for an identified object, in this example an instance of a chair.

[0159] In step 1810, the object identification module 214 may associate each pixel of the two-dimensional object detection with a surfel or surface of the three-dimensional model of the physical environment obtained by the estimated depth data from step 1806. In some embodiments, the object identification module 214 has associated a cluster of pixels of the two-dimensional object detection with a surfel or surface of the three-dimensional model of the physical environment obtained by the estimated depth data from step 1806. The estimated depth data may be used to determine the location of a particular two-dimensional object within the three-dimensional model. The object identification module 214 may identify a unique object within the three-dimensional model of the physical environment based on the location of each two-dimensional object detection.

[0160] In step 1814, the object identification module 214 may provide a cumulative confidence of a particular object identified in the collection of three-dimensional object detections based on the three-dimensional object probability. The cumulative confidence may be an aggregate confidence, an average confidence, a weighted average confidence, etc. The object identification module 214 may send a request to the object data store 220 to update or generate an object instance entry for a particular object. The object identification module 214 may send a request to the object data store 220 to update or generate an object type entry associated with a particular object type. Additionally, the object identification module 214 may send a request to the tag module 216 to define a tag and a particular point in the three-dimensional model to associate the tag. The object instance entry may include or indicate a particular instance of an object type. The object type entry may be associated with a particular object type. In some embodiments, the object data store 220 may store an object type count indicating a total number of objects of an object type on a floor of the 3D model, a portion of the 3D model, and / or the entire 3D model. The object identification module 214 may increase the total number of inputs of the object type associated with the dining room chair.

[0161] FIG. 21 shows a flow chart of an object detection process 2100 according to some embodiments. In step 2102, the communication module 204 receives a structured image collection from the image capture device 106A. The structured image collection may be generated as a result of a capture process from a Matterport Pro3 camera. The structured image collection generated by the Matterport capture process generates a depth map corresponding to each image in the image collection. In some embodiments, step 2102 includes color processing to reduce image artifacts. The structured image collection may include panoramic 2D images, 3D panoramic images, and depth maps of the physical environment.

[0162] In step 2104, the object identification module 214 includes Mask R-CNN, which detects and identifies two-dimensional objects in the digital image. The object identification module 214 may further identify image pixels associated with the two-dimensional objects. These identified pixels may be known as object pixels. In some embodiments, Mask R-CNN may provide a probability that a particular object in each of the subsets of two-dimensional object detections belongs to a particular category of two-dimensional objects.

[0163] An example of Mask R-CNN can be found at https: / / arxiv.org / abs / 1703.06870. Mask R-CNN takes a color image as input and generates a pixel "mask" and a set of classification "logits" for each object detected in the image. The classification "logits" are estimates of the likelihood that the object belongs to each of a fixed set of semantic categories of objects that the network has been trained to detect. In some embodiments, the object identification module 214 may generate a bounding polygon that indicates the boundary or outer extent of the identified object.

[0164] In step 2106, for each 2D object detected in step 2104, the object identification module 214 may associate object pixels with surfels or surfaces of the 3D model of the physical environment to obtain 3D object detections. The object identification module 214 may provide a probability that a particular object in each of the subset of 3D object detections belongs to a particular category of 3D objects.

[0165] In step 2108, the object identification module 214 may merge together three-dimensional object detections that correspond to the same object. In some embodiments, the object identification module 214 may identify and merge the three-dimensional object detections based on their respective spatial characteristics, for example, the location of each three-dimensional object detection in a three-dimensional model of the physical environment.

[0166] In addition to merging the 3D object detections, the object identification module 214 may further provide a cumulative confidence of a particular object belonging to a particular object category based on the 3D object probability of each 3D object detection.

[0167] In step 2110, the object identification module 214 provides the accumulated confidence to a user interface. An example of this can be seen in Figure 20. A user may interact with an example user interface 2000 that shows an icon that indicates an area of ​​the physical environment along with the spatial location where the object is located.

[0168] FIG. 22 is a block diagram illustrating an exemplary machine entity capable of reading instructions from a machine-readable medium and executing those instructions within a processor that performs the machine processing tasks discussed herein, such as the engine operations described above. In particular, FIG. 22 illustrates a diagrammatic representation of a machine in the exemplary form of a computer system 2200 within which instructions 2224 (e.g., software) may be executed that cause the machine to perform any one or more of the methods discussed herein. In alternative embodiments, the machine may operate as a standalone device or may be connected (e.g., networked) to other machines, for example, via the Internet. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or may operate as a peer machine in a peer-to-peer (or distributed) network environment.

[0169] The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cell phone, a smart phone, a web appliance, a network router, a switch or bridge, or any machine capable of executing (sequentially or otherwise) instructions 2224 that specify actions to be taken by the machine. Additionally, although only a single machine is depicted, the term "machine" shall be interpreted to include any collection of machines that individually or jointly execute instructions 2224 to implement any one or more of the methods discussed herein.

[0170] The exemplary computer system 2200 includes a processor 2202 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), a main memory 2204, and a static memory 2206, which are configured to communicate with each other via a bus 2208. The computer system 2200 may further include a graphics display unit 2210 (e.g., a plasma display panel (PDP), a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). Computer system 2200 may also include an alphanumeric input device 2212 (e.g., a keyboard), a cursor control device 2214 (e.g., a mouse, trackball, joystick, motion sensor, or other pointing device), a data store 2216, a signal generating device 2218 (e.g., a speaker), an audio input device 2226 (e.g., a microphone), and a network interface device 2220, which are configured to communicate via bus 2208.

[0171] The data store 2216 includes a machine-readable medium 2222 having instructions 2224 (e.g., software) stored thereon, the instructions 2224 implementing one or more of the methods or functions described herein. The instructions 2224 (e.g., software) may reside, completely or at least partially, within the main memory 2204 or within the processor 2202 (e.g., within a processor cache memory) during execution of the instructions 2224 by the processor 2202, which together constitute the computer system 2200, the main memory 2204, and the like, the machine-readable medium. The instructions 2224 (e.g., software) may be transmitted or received over a network (not shown) via a network interface 2220.

[0172] While in one exemplary embodiment machine-readable medium 2222 is shown to be a single medium, the term "machine-readable medium" should be interpreted to include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) capable of storing instructions (e.g., instructions 2224). The term "machine-readable medium" should also be interpreted to include any medium capable of storing instructions (e.g., instructions 2224) for a machine to execute and that cause a machine to perform any one or more of the methods disclosed herein. The term "machine-readable medium" should include, but is not limited to, data repositories in the form of solid-state memories, optical media, and magnetic media.

[0173] As used herein, the term "module" refers to computational logic that provides a particular function. Modules may be implemented in hardware, firmware and / or software. When modules described herein are implemented as software, they may be implemented as stand-alone programs, but may also be implemented through other means, such as as part of a larger program, as any number of separate programs, or as one or more statically or dynamically linked libraries. It will be understood that the named modules described herein represent one implementation, and that other embodiments may include other modules. Additionally, other embodiments may lack modules described herein and / or distribute the described functionality among modules in various ways. Additionally, functionality that contributes to more than one module may be incorporated into a single module. In one implementation, when modules are implemented by software, they are stored on a computer-readable persistent storage device (e.g., hard disk), loaded into memory, and executed by one or more processors as described above with respect to FIG. 22. Alternatively, hardware or software modules may be stored elsewhere in the computing system.

[0174] As referred to herein, a computer or computing system includes hardware elements used for the operations described herein, whether or not specifically mentioned in FIG. 22, including, for example, one or more processors, high speed memory, hard disk storage and backup, network interfaces and protocols, input devices for data entry, and output devices for display, printing or other representation of data. Many variations from the system architecture are possible. Such system entities and their respective functions may be combined or redistributed.

Claims

1. 1. A system comprising: one or more processors; Memory and The memory comprises: receiving a plurality of images representing an interior of a physical environment; using a neural network for object recognition to identify a first object in a first image of the plurality of images associated with location information of the first object indicative of a location relative to the physical environment; using the object recognition neural network to identify a second object in a second image of the plurality of images; determining whether the first object in the first image and the second object in the second image are located near or at a similar location in the physical environment based on location information associated with the first image and location information of a second object associated with the second image; determining that the first object and the second object are a single object instance if the first object and the second object are located near or at a similar location; storing said similar locations associated with said single object; displaying an interactive walk-through visualization of a 3D model of the physical environment including the single object; receiving an object information request regarding a location of an object through the interactive walk-through visualization; Providing similar locations of the single object for display in the interactive walk-through visualization. and instructions for controlling the one or more processors to:

2. The system of claim 1 , wherein the instructions further control the one or more processors to determine whether the first object and the second object are of the same object type.

3. 3. The system of claim 2, wherein the first object and the second object are the single object instance if the first object and the second object are located near or at a similar location and the first object is of the same object type as the second object.

4. 3. The system of claim 2, wherein identifying the first object using the object recognition neural network further comprises using the neural network to categorize the first object and identify an object type of the first object.

5. The system of claim 1 , wherein the instructions further control the one or more processors to determine location information of the first object and location information of the second object based on a mesh of a 3D model.

6. 2. The system of claim 1 , wherein the first image includes a position and orientation of an image capture device that captured the first image within the physical environment, and the instructions further control the one or more processors to determine location information of the first object based on the position and the orientation of the image capture device that captured the first image.

7. 7. The system of claim 6, wherein the second image includes a position and orientation of an image capture device that captured the second image within the physical environment, and the instructions further control the one or more processors to determine location information of the second object based on the position and the orientation of the image capture device that captured the second image.

8. 2. The system of claim 1, wherein the first image is associated with depth data of the physical environment, the depth data being generated by a depth data device, and the location of the first object is determined based on the first object shown in the first image and the associated associated depth data.

9. The system of claim 8 , wherein the depth data device is a LiDAR.

10. A non-transitory computer-readable medium comprising executable instructions, the executable instructions being executable by one or more processors to perform a method, the method comprising: Receiving a plurality of images representative of an interior of a physical environment; using a neural network for object recognition to identify a first object in a first image of the plurality of images, the first object being associated with first object location information, the first object location information indicating a location relative to the physical environment; using the object recognition neural network to identify a second object in a second image of the plurality of images; and determining whether the first object in the first image and the second object in the second image are located near or at a similar location in the physical environment based on location information associated with the first image and location information of a second object associated with the second image; determining that the first object and the second object are a single object instance if the first object and the second object are located near or at a similar location; storing the similar locations associated with the single object; and displaying an interactive walk-through visualization of a 3D model of the physical environment, the 3D model including the single object; receiving an object information request regarding a location of an object through the interactive walk-through visualization; providing similar locations of the single object for display in the interactive walk-through visualization; and A non-transitory computer readable medium comprising:

11. The executable instructions may further include: Determining whether the first object and the second object are of the same object type.

11. The non-transitory computer-readable medium of claim 10, executable by the one or more processors to:

12. 12. The non-transitory computer-readable medium of claim 11, wherein the first object and the second object are the single object instance if the first object and the second object are located near or at a similar location and the first object is of the same object type as the second object.

13. 12. The non-transitory computer-readable medium of claim 11, wherein identifying the first object using the object recognition neural network further comprises using the neural network to categorize the first object and identify the object type of the first object.

14. The executable instructions may further include: and further controlling the one or more processors to determine location information of the first object and location information of the second object based on a mesh of a 3D model.

11. The non-transitory computer-readable medium of claim 10, executable by the one or more processors to:

15. 11. The non-transitory computer-readable medium of claim 10, wherein the first image includes a position and orientation of an image capture device that captured the first image within the physical environment, and the executable instructions are further executable by the one or more processors to further control the one or more processors to determine location information of the first object based on the position and the orientation of the image capture device that captured the first image.

16. 16. The non-transitory computer-readable medium of claim 15, wherein the second image includes a position and orientation of an image capture device that captured the second image within the physical environment, and the executable instructions are further executable by the one or more processors to determine location information of the second object based on the position and the orientation of the image capture device that captured the second image.

17. 11. The non-transitory computer-readable medium of claim 10, wherein the first image is associated with depth data of the physical environment, the depth data being generated by a depth data device, and the location of the first object is determined based on the first object shown in the first image and the associated associated depth data.

18. 20. The non-transitory computer-readable medium of claim 17, wherein the depth data device is a LiDAR.

19. 1. A method, comprising: Receiving a plurality of images representative of an interior of a physical environment; using a neural network for object recognition to identify a first object in a first image of the plurality of images, the first object being associated with first object location information, the first object location information indicating a location relative to the physical environment; using the object recognition neural network to identify a second object in a second image of the plurality of images; and determining whether the first object in the first image and the second object in the second image are located near or at a similar location in the physical environment based on location information associated with the first image and location information associated with the second image; determining that the first object and the second object are a single object instance if the first object and the second object are located near or at a similar location; storing the similar locations associated with the single object; and displaying an interactive walk-through visualization of a 3D model of the physical environment, the 3D model including the single object; receiving an object information request regarding a location of an object through the interactive walk-through visualization; providing the similar locations of the single object for display in the interactive walk-through visualization; and A method comprising:

Images