Target detection and interactive 3D model system and method

Automatically identify and classify targets in 3D models through neural networks and deep data, solving the problem of time-consuming and error-prone manual tagging, achieving efficient and accurate collection and display of target information, and improving the user experience of interactive visualization of 3D models.

CN118946909APending Publication Date: 2024-11-12COSTA REAL ESTATE INFORMATION CO
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202380026284.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-24
Filing Date
2023-03-10
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

In existing 3D models, manual marking of object detection and information collection is time-consuming and error-prone and difficult to scale, especially when the number of images increases, the lack of automated identification and marking technology leads to inefficiency.

Method used

Using neural network recognition and classification technology, we use multiple images to identify target positioning information, and provide target information in an interactive 3D visual environment, and use deep data such as LiDAR to generate 3D models to achieve automatic target detection and information collection.

Benefits of technology

It realizes automatic identification and classification of targets in 3D models, reduces the need for manual tagging, improves efficiency and accuracy, can quickly respond to user queries, and provides interactive target information display.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118946909A_ABST
    Figure CN118946909A_ABST
Patent Text Reader

Abstract

A system includes a processor and a memory containing instructions to control the processor to: receive an image representing an interior of a physical environment; using a neural network for target recognition to recognize a target in the image, the target being associated with a location relative to the physical environment; using a neural network for target recognition to identify another target in another image, determining whether the target in the image is located at a nearby or similar location based on location information associated with the target; if the target is located nearby or at a similar location, the target is an instance of a single target; in one embodiment, a method includes receiving a single target, storing a similar location associated with the single target, displaying an interactive sightseeing visualization containing a three-dimensional model of a physical environment of the single target, receiving a request for target location through the interactive sightseeing visualization, and providing the similar location of the single target for display in the interactive sightseeing visualization.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the current invention relate to interactive 3D visualization to provide information about a 3D modeled environment and more generally to automated systems for object detection, object information collection, and interactive response to user queries about 3D models and objects contained therein. Background Art

[0002] Three-dimensional (3D) visualizations and walkthroughs typically enable a user to view and / or engage with a 3D model of a given environment. Visualizations of 3D models of physical environments, such as houses, are becoming common. In one example, a user can access a 3D model through a website, such as a real estate website, and walk through the 3D model of a house. However, these 3D models are limited to walkthroughs and certain graphical elements that enable the user to navigate in the space. The 3D models do not identify objects in the model, nor do they allow the user to request information about the physical environment. Summary of the invention

[0003] An example system includes one or more processors and a memory, the memory containing instructions to control the one or more processors to: receive multiple images representing an interior of a physical environment; use a neural network for object recognition to recognize a first object in a first image of the multiple images, the first object being associated with first object location information, the first object location information indicating a location relative to the physical environment; use the neural network for object recognition to recognize a second object in a second image of the multiple images, based on the location information associated with the first image and the second object location information associated with the second image, determine whether the first object in the first image and the second object in the second image are located near or at a similar location in the physical environment; and if the first object and the second object are located near or at a similar location, the first object and the second object are instances of a single object; store the similar location associated with the single object, display an interactive tour visualization of a 3D model of the physical environment, the 3D model including the single object, receive a request for object information about the location of the object through the interactive tour visualization, and provide the similar location of the single object for display in the interactive tour visualization.

[0004] In various embodiments, the instructions further control one or more processors to determine whether the first target and the second target are of the same target type. In one example, if the first target and the second target are located in a nearby or similar position, and the first target and the second target are of the same target type, then the first target and the second target are instances of a single target. In another example, identifying the first target using a neural network for target recognition also includes classifying the first target using a neural network to identify the target type of the first target. In an example system, the instructions further control the one or more processors to identify and determine the first target location information and the second target location information based on a mesh of a 3D model. In some embodiments, the first image includes a position and orientation of an image capture device that captured the first image in a physical environment, and the instructions further control the one or more processors to determine the first target location information based on the position and orientation of the image capture device that captured the first image.

[0005] In one example system, the second image includes a position and orientation of an image capture device in the physical environment that captured the second image, and the instructions further control the one or more processors to determine second target location information based on the position and orientation of the image capture device that captured the second image. In one example, the first image is associated with depth data of the physical environment, the depth data is generated by a depth data device, and the first target location is determined based on the first target depicted in the first image and the associated related depth data. The system of claim 8, wherein the depth data device is a laser radar (LiDAR).

[0006] An example of a transient computer-readable medium containing executable instructions that can be executed by one or more processors to implement a method, the method comprising: receiving a plurality of images representing an interior of a physical environment; using a neural network for object recognition to recognize a first object in a first image of the plurality of images, the first object being associated with first object location information, the first object location information indicating a location relative to the physical environment; using the neural network for object recognition to recognize a second object in a second image of the plurality of images, based on the location information associated with the first image and the second object location information associated with the second image, determining whether the first object in the first image and the second object in the second image are located near or at a similar location in the physical environment; if the first object and the second object are located near or at a similar location, the first object and the second object are instances of a single object; storing the similar location associated with the single object, displaying an interactive tour visualization of a 3D model of the physical environment, the 3D model including the single object, receiving a request for object information regarding the location of the object through the interactive tour visualization, and providing the similar location of the single object for display in the interactive tour visualization.

[0007] An example method includes: receiving multiple images representing an interior of a physical environment; using a neural network for object recognition to recognize a first object recognized in a first image of the multiple images, the first object being associated with first object location information, the first object location information indicating a location relative to the physical environment; using the neural network for object recognition to recognize a second object recognized in a second image of the multiple images, and based on the location information associated with the first image and the location information associated with the second image, determining whether the first object in the first image and the second object in the second image are located in the vicinity or at a similar location in the physical environment; if the first object and the second object are located in the vicinity or at a similar location, the first object and the second object are instances of a single object; storing the similar location associated with the single object, displaying an interactive tour visualization of a 3D model of the physical environment, the 3D model including the single object, receiving a request for object information regarding the location of the object through the interactive tour visualization, and providing the similar location of the single object for display in the interactive tour visualization.

[0008] Another example system includes: one or more processors, a memory, the memory containing instructions for controlling the one or more processors to: identify a first view of a first target in a first two-dimensional image, the first view being a first set of pixels associated with a first position and a first orientation of an image capture device when the image capture device captures the first two-dimensional image; identify a second view of a second target in a second two-dimensional image, the second view being a second set of pixels associated with a second position and a second orientation of the image capture device when the image capture device captures the second two-dimensional image; map the first set of pixels to a position in a 3D model based on the first position and the first orientation of the image capture device, the 3D model being a representation of a physical environment; determine that the first target and the second target are the same physical target based on the mapping of the second set of pixels, the second position, and the first set of pixels in the second two-dimensional image, and use at least some of the first set of pixels and the second set of pixels to represent the same physical target in the 3D model of the physical environment.

[0009] In some embodiments, determining that the first target and the second target are the same physical target is also based on the depth map of the physical environment. In one example, the first target position of the same physical target in the 3D model is determined based on the depth map of the physical environment. In one example, the first pixel set is a first surface element set, and the second pixel set is a second surface element set. In various embodiments, it also includes identifying the target type of the same physical target based on at least some of the first pixel set. In some embodiments, the same physical target is further classified using CNN based on at least some pixels in the second pixel set to identify the target type of the same physical target. In another example, the example system also includes generating a first sign associated with the target type and the position of the same physical target in the 3D model. The example system also includes: identifying a third view of a third target in a third two-dimensional image, the third view is a third pixel set associated with a third position and a third direction of the image capture device when the image capture device captures the third two-dimensional target, and determining that the third target is a physical target different from the first and second targets based on the mapping of the third position and the first pixel set. An example system also includes identifying the target type of the third target, and the target type of the third target is the target type of the first target. In various embodiments, the example system further includes: generating a second sign associated with the third target, and incrementing a count of the target type indicating the number of target types represented in the 3D model. Another example system further includes: storing the first sign and the second sign in a data structure, receiving a query regarding the location of targets of the target type within the 3D model, responding to the query based on information in the data structure, including the first sign, the second sign, a first target location of the first target in the 3D model and a second target location of the third target in the 3D model, the first target location of the same physical target and the second target location of the third target based on one or more depth maps of the physical environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 Depicted is a block diagram of an example environment capable of receiving a digital image of a physical environment and automatically detecting objects depicted in the digital image and providing information about the detected objects.

[0011] Figure 2 Depicted is a block diagram of an object detection and localization system in accordance with some embodiments.

[0012] Figure 3 Depicted is a flow diagram of an object detection process in accordance with some embodiments.

[0013] Figure 4 is an example of an example user interface for identifying the locations of multiple detected objects of similar types in accordance with some embodiments.

[0014] Figure 5A flow chart is depicted of a multiple capture position method of an object detection process in accordance with some embodiments.

[0015] Figure 6 is a flow chart for recognizing the same or different instances of an object in a two-dimensional image based on the position of an image capture device that captured the image, in some embodiments.

[0016] Figure 7-10 Depicts an example of a collection of images of the same physical space viewed by multiple image capture devices, with 2D object detection highlighted.

[0017] Fig.11 Depicted Figure 7-10 The image collection is merged with the positioning of the image capture device and the positioning of the two-dimensional detection target.

[0018] Fig.12 Depicted is a flow diagram for collecting information and providing interaction with a 3D model in some embodiments.

[0019] Fig.13 Depicted is a flow chart of a grid method for an object detection process in accordance with some embodiments.

[0020] Fig.14 Describes an overview of the implementation of the object detection and localization system.

[0021] Fig.15A Depicted are components that make up a mesh of objects in a physical environment, according to some embodiments.

[0022] Fig. 15B Depicted are the locations, sizes, and shapes of two identified objects in a physical environment, according to some embodiments.

[0023] Fig.16A Depicted are components that make up a mesh of objects in a physical environment, according to some embodiments.

[0024] Fig. 16B Depicted are the location, size, and shape of identified objects in a physical environment, highlighted in different colors, according to some embodiments.

[0025] Figure 17A-17C Depicted is the localization of the same detection target from three different viewpoints in accordance with some embodiments.

[0026] Fig.18 A flow chart of a depth estimation method for an object detection process according to some embodiments is depicted.

[0027] Fig.19 Depicted are examples of two-dimensional color images and corresponding two-dimensional depth mathematics in accordance with some embodiments.

[0028] Fig. 20 is an example user interface showing recognized objects with labels corresponding to three-dimensional points in physical space in accordance with some embodiments.

[0029] Fig.21 Depicted is a flow diagram of an object detection process in accordance with some embodiments.

[0030] Fig. 22 Depicted is a block diagram of an example digital device in accordance with some embodiments. DETAILED DESCRIPTION

[0031] Various embodiments described herein can collect information about different objects within a 3D model. A user can request information (e.g., by providing a query) about a 3D model (e.g., through interaction with a visualization or tour and / or an API). In response, the various systems discussed herein can retrieve relevant information about the objects in the 3D model and respond to the request by providing text, images, and / or changes to the user's visualization of the 3D model to provide the information.

[0032] In order to be able to collect object information (e.g., the location of the object in the 3D model, the object type, the number of objects of the same object type, etc.), it is necessary to identify the objects appearing in the 3D model and collect information about these objects. Historically, identifying objects depicted in images has been performed manually. For example, a person may review one image at a time to manually label and / or identify the objects. The person may also manually provide descriptive metadata for the objects.

[0033] Unfortunately, manual labeling of images is time-consuming, expensive, and error-prone, especially when the collection of images to be labeled is large. Therefore, manual labeling of images is not scalable as the number of images increases and / or the number of items to be labeled in the images also increases.

[0034] To avoid the human effort required for manual tagging, limited technology has been developed to automatically identify and tag digital images with certain types of information. For example, digital cameras can automatically store certain types of information for each image, such as the time, date, and GPS coordinates when the image was captured.

[0035] Various embodiments described herein include systems and methods for automatically identifying objects that appear or will appear in a 3D model, classifying objects (e.g., labeling), identifying the location of objects in a 3D model, collecting information about the objects, and storing associated information. An interactive visualization of a 3D model (e.g., a tour interface that enables a user to tour the 3D model) can be provided, allowing the user to request additional information about the objects in the visualization (e.g., through a query field or interactive GUI element) (e.g., "How many electrical outlets are there in a house?"). The system can retrieve the collected information about the identified objects and provide a response to the request (e.g., through text, visual elements, etc.).

[0036] It should be understood that there are many ways to identify objects in an image (e.g., using multiple images, using depth information, or using a grid), and there are also many ways to collect object metadata (e.g., the type of object, the number of object types per floor or building, the location of one or more objects, etc.).

[0037] The various systems and methods described herein allow a scalable system to automatically detect objects, differentiate between objects (e.g., identifying different chairs on different floors), and collect the location and other information of each object. Performing these steps manually can be error-prone, slow, and computationally inefficient.

[0038] Figure 1 A block diagram of an example environment 100 is depicted. Environment 100 includes a communication network 102, environment images 104A and 104B (individually, collectively, environment images 104), image capture devices 106A and 106B (individually, collectively, image capture devices 106), a building 108, a building 110, an object detection and localization system 112, a user system 114, and a model data store 116.

[0039] In this example, the target detection and positioning system 112 can provide or enable visualization and / or 3D model to allow the user to tour the modeled environment. The target detection and positioning system 112 can provide interactive visualization and / or options to allow the user to request information about one or more targets depicted in the 3D model. The target detection and positioning system 112 can retrieve information to provide a response. In various embodiments, the target detection and positioning system 112 can automatically identify targets from the 2D image and / or 3D model, collect information associated with the target (e.g., the type of target identified, the number of targets of a particular type depicted in the 3D model or a portion of the 3D model, the location of the target in the modeled environment, etc.).

[0040] In some embodiments, the communication network 102 represents one or more computer networks (e.g., LAN, WAN, etc.). The communication network 102 can provide communication between environments, such as the image capture device 106, the target detection and positioning system 112, the user system 114, and the model data storage 116. In some implementations, the communication network 102 includes computer devices, routers, cables, uses, and / or other network topologies. In some embodiments, the communication network 102 can be wired and / or wireless. In various embodiments, the communication network 102 can include the Internet, one or more networks, which can be public, private, IP-based, non-IP-based, and the like.

[0041] In various embodiments, environmental images 104A include digital images of a physical environment, such as the interior of building 108. These images may be captured by placing one or more image capture devices 106A at different locations within building 108. In some embodiments, environmental images 104A include digital images of the exterior of building 108. Environmental images 104A may depict enough of the interior of building 108 (e.g., the living spaces on each floor) that these images may serve as the basis for creating a 3D model of the interior of building 108.

[0042] The digital images or videos captured by the image capture device 106A may be sent to the target detection and localization system 112. The image capture device 106 may transmit the digital images to the target detection and localization system 112. In various embodiments, the digital images of the interior and / or exterior of a particular location may be transmitted to the model data store 116. Alternatively, the image capture device 106 may be wirelessly coupled to a smart device, and the smart device may provide the digital images to the target detection and localization system 112. In some embodiments, the images may be downloaded to a card or other media for later upload to the target detection and localization system 112.

[0043] Image capture device 106 may include sensors and / or software for identifying position and / or direction. In various embodiments, image capture device 106 may associate the position and / or direction of image capture device 106 with one or more images captured by image capture device 106 at the position and / or direction. In one example, the position of image capture device 106 may be provided by a GPS sensor or any other system for assisting positioning that provides GPS coordinates. The direction of image capture device may be identified based on the position of image capture device and the field of view of the lens from image capture device. The direction and / or position of image capture device 106 may be determined or identified in any number of ways.

[0044] In some embodiments, the image capture device 106 is a complementary metal oxide semiconductor (CMOS) image sensor (e.g., a Sony IMX283 ~20 megapixel CMOS MIPI sensor with an NVIDIA Jetson Nano SOM). In various embodiments, the image capture device is a charge coupled device (CCD). In one example, the image capture device is a red green blue (RGB) sensor. In one embodiment, the image capture device 106 is an infrared (IR) sensor. The image capture device 106 may include a lens assembly to provide a wide field of view to the image capture device.

[0045] In some embodiments, the image capture device 106 may include a depth sensor (such as LiDAR, SPAD, or structured light) to obtain depth data. Depth data can be defined as the distance between a point in the physical environment depicted in a pixel of an image captured by the image capture device and the image capture device. Alternatively, in other embodiments, multiple image sensors may be used to obtain depth data, such as stereo-assisted imaging, in which multiple image sensors are offset by a predetermined distance. These multiple image sensors can capture substantially the same physical environment with a slight offset. The processor can use the digital images captured by these multiple image sensors to create or enhance the illusion of depth in the form of a three-dimensional image.

[0046] In other embodiments, the depth data may be acquired by a LiDAR device ( Figure 1 It should be understood that depth data from a LiDAR sensor (e.g., as part of image capture device 106 or separate therefrom) may not be available.

[0047] In various embodiments, the depth data defines depth and / or positioning information about the scanned physical environment. The depth data may be associated with the positioning of objects, walls, floors, ceilings, etc., which may be the subject of images captured by the image capture device(s).

[0048] In some embodiments, object detection and localization system 112 may receive any number of two-dimensional images of the physical environment from image capture device(s) 106 .

[0049] The object detection and localization system 112 may utilize different methods or approaches to recognize and classify objects depicted in the two-dimensional image. The method or approach used by the object detection and localization system 112 may depend on the type of two-dimensional image received by the object detection and localization system 112. The different processes may include, but are not limited to, recognizing objects based in part on the orientation and position of the image capture device 106 that captured the image, using depth data (e.g., depth data generated from an image or a LiDAR system), or from the orientation and position of a grid.

[0050] In one example, the target detection and localization system 112 receives a two-dimensional image (and optionally depth data) of a physical environment from the image capture device 106. In this approach, targets are identified or detected in two dimensions in a subset of the two-dimensional image (e.g., using a neural network such as the RMask CNN). The target (e.g., pixels associated with the identified target) can be mapped to a three-dimensional surface or location of the physical environment. A collection of images of a particular identified target can be identified based on the mapping, and once the collection of images of a particular target is identified, image information (e.g., pixels) from different viewpoints of the same physical target can be applied to a 3D model visualization to depict the target (e.g., in one or more viewpoint directions).

[0051] In some embodiments, the object detection and localization system 112 trains a neural network or uses a trained neural network to recognize a specific type of object depicted in a two-dimensional image. Once trained, the neural network can recognize objects from images captured by one or more image capture devices 106. An object type is a specific type of object or a set of object types. For example, an object type can be "chair", "table", "fireplace", "outlet", "toilet", "shower", "sink", etc. An object instance is a specific object of that object type. For example, a chair located in a room in front of a fireplace is an instance of a chair, with an object type of "chair", and another different chair on a different floor is a different instance of a chair (i.e., there are two chair instances in this example), both of which are of the "chair" object type, even though the chairs may be completely different from each other (e.g., one may be a kitchen table chair, while the other may be a recliner).

[0052] In some embodiments, the neural network can locate each pixel of each object in the image, for example, using instance segmentation (e.g., using Mask R-CNN or any R-CNN process). For each image, the object detection and localization system 112 can create region proposals, classify the region proposals, and generate segmentation masks to identify specific objects for recognition (e.g., labeling and / or annotation), counting, and / or localization.

[0053] In various embodiments, the object detection and localization system 112 provides (e.g., streams and / or outputs) all or part of the 3D model to a digital device (e.g., a user system 114 or a remote website). The user may utilize the user system 114 to view all or part of the 3D model (e.g., Fig.19 Example two-dimensional panoramic image 1900).

[0054] In some embodiments, the object detection and localization system 112 may provide the user system 114 with a visual representation of an area of ​​a floor plan of a building (ie, a visualization of a 3D model). Figure 4 4 is an example user interface 400. A user of user system 114 can interact with area 410 to view and navigate around that portion of a building. In some embodiments, a user can interact with list 420, which provides multiple instances of targets in a given target type (ie, target category).

[0055] In various embodiments, the example user interface 400 provides an interactive tour of a 3D model (e.g., a tour of a 3D model of a physical environment). Figure 4 In the example of FIG. 4 , list 420 lists identified outlets located in the 3D model. A user can interact with a specific outlet, such as entry 422. In response, user interface 400 can provide the location of the specific outlet in the floor plan of the building.

[0056] In some embodiments, the example user interface 400 may depict a dollhouse view of a tour view of a 3D model. In response to a user's query, the example user interface may retrieve information and / or provide a response to the user. In some embodiments, the visualization module 218 may indicate locations in the 3D model to indicate targets of a target type indicated by the query. For example, in Figure 4 In response to the query, the visualization module 218 can highlight, represent, annotate, etc. the location of each socket on the 3D map. It should be understood that the visualization module 218 can indicate the target of interest in any number of ways (e.g., by pulsing, lighting, highlighting, arrows, etc.).

[0057] In some embodiments, user system 114 is a digital device that can communicate with other digital devices and systems. A digital device is any device with a processor and memory. In some embodiments, user system 114 can be or include one or more mobile devices (e.g., smart phones, cellular phones, smart watches, tablet computers, etc.), desktop computers, notebook computers, etc. In some embodiments, a user can interact with user system 114 using, for example, a web browser or a mobile application to communicate with target detection and positioning system 112.

[0058] In some embodiments, the object detection and localization system 112 provides an image to the user system 114 that represents the location of the object detected by the object detection and localization system 112. In various embodiments, the object detection and localization system 112 may provide a label or tag that includes physical or semantic information about the object, such as an object class or a property of the detected object. Fig. 20 An example of this is shown in A user may interact with an example user interface 2000 that depicts an area of ​​a physical environment and an icon that depicts a location space where a target is located.

[0059] In some embodiments, the model data store 116 can be any one and / or multiple structures suitable for the captured data, such as 3D models, LiDAR data, images, etc. In some examples, the model data store 116 is an active database, a relational database, a self-referencing database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, etc. The model data store 116 can store digital images or videos captured by the image capture device 106. In some embodiments, the model data store 116 can store three-dimensional models of the interior and / or exterior of various physical environments. The three-dimensional model can be created by the target detection and localization system 112. In one example, the three-dimensional model can be created by a third-party software application (not shown).

[0060] Figure 2 A block diagram of the target detection and localization system 112 is depicted in accordance with some embodiments. The target detection and localization system 112 includes a communication module 202, a communication module 204, a geolocation module 206, a depth data module 210, an alignment module 208, a mesh construction module 212, a target recognition module 214, a labeling module 216, a visualization module 218, a target data store 220, and an image data store 222.

[0061] The communication module 202 may be connected to any module or data storage device of the target detection and positioning system 112. Figure 1 Send and request requests, images or data between devices in environment 100.

[0062] The communication module 204 may receive any number of two-dimensional images captured by the image capture device 106 of a physical environment (eg, the interior of a building, facility, house, etc.) In some embodiments, the communication module 204 receives depth data of the physical environment from the image capture device 106 .

[0063] In one example, two-dimensional images captured by any number of image capture devices 106 of a premises interior (e.g., a physical environment) are provided to the object detection and localization system 112. The two-dimensional images can be received by the communication module 204. In some embodiments, the communication module 204 can receive depth data of the premises interior from a depth sensor (e.g., the depth sensor is part of the image capture device 106 or separate from the image capture device 106).

[0064] In some embodiments, the geolocation module 206 can communicate with an external positioning system such as a global positioning system (GPS). The geolocation module 206 can match one or more raw GPS coordinates (e.g., captured by the image capture device 106 or a digital device when the image was captured) to a single capture location based on the time at which the image data was captured at the capture location and the time at which the GPS measurements were captured.

[0065] For example, in an implementation where image data is captured at one or more capture locations by rotating the image capture device around a fixed axis, the two-dimensional image can be time stamped with information indicating the start time and the end time of the rotation. According to this example, the geolocation module 206 can identify GPS coordinates captured between the start and end times of the rotation and associate these GPS coordinates with the capture locations.

[0066] In other implementations, the geolocation module 206 can be configured to associate raw GPS coordinates with a particular capture location based on GPS coordinates within a defined time range relative to the time the image was captured at the capture location. In this regard, the geolocation module 206 can group or cluster sets of raw GPS coordinates with a single capture location based on timestamp information associated therewith. Based on clustering the raw GPS coordinates based on matching the capture time of the raw GPS coordinates with the capture time / period associated with the corresponding capture location, in some implementations, the geolocation module 206 can also delete any GPS coordinates captured at a time that does not correspond to the image capture time (e.g., delete GPS coordinates captured between capture locations).

[0067] In various embodiments, the geolocation module 206 may reside on the image capture device 106, or on a digital device capable of acquiring GPS coordinates that is local to the image capture device 106. In this example, the object detection and location system 112 may not include the geolocation module 206.

[0068] In some embodiments, two or more two-dimensional images received from the image capture device 106 require alignment from the alignment module 208 to generate a two-dimensional panoramic image (up to 360 degrees) of the physical environment. In one example, the alignment module 208 aligns two-dimensional images (e.g., panoramic images) and depth data of the same physical environment to generate a three-dimensional model of the physical environment. In some embodiments, the object recognition module 214 recognizes a three-dimensional object from the three-dimensional model.

[0069] The depth data module 210 can determine depth data associated with the two-dimensional image received by the object detection and positioning system 112. In some embodiments, the depth data module 210 receives the depth data from the image capture device 106 (or a depth device such as LiDAR) in the form of a time delay between the detection of an outgoing light pulse and a reflected light pulse. The time delay determines the distance from the depth sensor to a specific point in the physical environment.

[0070] In various embodiments, the depth data module 210 resides on the image capture device 106, or on a device local to the image capture device 106 when the depth data is captured. In some embodiments, the object detection and localization system 112 may not include the depth data module 210.

[0071] The object detection and localization system 112 may utilize the two-dimensional images and the depth data to generate a three-dimensional model (eg, a 3D model) of the physical environment (eg, the interior of a house).

[0072] In some embodiments, the depth data module 210 includes a machine learning algorithm (e.g., a neural network) that uses two or more two-dimensional images of the physical environment to estimate depth data for all or a portion of the scanned physical environment. The machine learning algorithm can use characteristics of the two-dimensional image, such as color or light intensity, to determine the distance of a pixel in the image from the image capture device. In the case where the image capture device 106 includes multiple image sensors, such as stereo-assisted imaging, where the multiple image sensors are offset by a predetermined distance, the depth data module 210 can use the offset images of the physical environment to determine the depth data of the environment captured in one or more images.

[0073] In some embodiments, before a three-dimensional model of the physical environment can be generated, a mesh of the environment is created by the mesh construction module 212. The mesh may include a series of triangles, lines, surfaces (e.g., non-uniform rational basis splines ("NURBS")), quadrilaterals, n-tuples, or other geometric shapes that can connect a collection of points. In one aspect, portions of the mesh may include image data describing texture, color, intensity, etc. In various embodiments, a captured 2D panoramic image (or portion thereof) may be overlaid on portions of the mesh to create a textured mesh. The textured mesh may include walls, floors, ceilings, and various objects within the space. The textured mesh may be dimensionally accurate, allowing a user of the user interface to determine the physical distance between two points in the three-dimensional model of the physical environment. In one example, the mesh construction module 212 utilizes a method for creating a mesh, such as Laplacian smoothing, parallel mesh generation, or a stretched mesh method.

[0074] In one example, the mesh is a triangular mesh, where pixels in a particular triangle of the triangular mesh represent areas of the environment on a common plane. Thus, the subject digital 3D model may be generated based on 2D image data, 2D sensory data, sensory data combined with raw 2D data, 3D spatial data (e.g., spatial depth and distance information), computer-generated position data, etc. In some embodiments, the three-dimensional model of the physical environment is generated without using a mesh.

[0075] In some embodiments, the object recognition module 214 includes a machine learning (ML) algorithm trained to recognize objects captured in one or more images of the physical environment. In one example, the object recognition module 214 recognizes multiple objects from different two-dimensional images of the physical environment. The objects captured by the two-dimensional images can represent instances of the physical object at one or more specific viewpoints. Each instance of the object can be associated with a three-dimensional surface in physical space, which indicates the location within the 3D model.

[0076] Each 3D object detection may not be a unique object in the physical environment. It should be understood that the same object may be captured in different images from different viewpoints. For example, Fig.17A , Fig. 17B and Fig. 17C The three-dimensional object detections 1712, 1722, and 1732 correspond to the same chair from different viewpoints.

[0077] Each individual object identified from the image may correspond to an instance of a specific object in the three-dimensional model of the physical space. Each instance of a specific object may be classified by the object recognition module 214. Different instances of the same object may be classified as different objects. Aggregate confidence may be used to classify a specific object using a separate category for each different instance of the same object.

[0078] For example, both objects can be classified as "chairs", even though the objects are different from each other (e.g., one is a kitchen chair and the other is a rocking chair). This classification may be referred to as an "object type" in this article (e.g., two instances have the same "object type", i.e., "chair"). In this example, the kitchen chair and the rocking chair are different instances (i.e., different object instances).

[0079] In one or more embodiments, the target recognition module 214 can be configured to use hardware, software, or a combination of hardware and software that provides automatic detection of visual targets that appear in an image, video, or 3D model to recognize targets (e.g., structural targets, fixtures, appliances, furniture, decorative targets, people, places, things, etc.) that appear in the 3D model. For example, with respect to a 3D model that depicts a home interior space, the target recognition module 214 can recognize walls, floors, ceilings, windows, fixtures, appliances, furniture, etc. The target recognition module 214 can also recognize target groups that are associated with each other. For example, the target recognition module 214 can recognize target groups associated with a particular room (e.g., a kitchen or bedroom). The target recognition algorithm relies on matching, machine learning, or a pattern recognition algorithm that uses appearance-based or feature-based techniques. Common techniques include edge detection, gradients, histograms of oriented gradients (HOG), Haar wavelets, and linear binary patterns. In various embodiments, the object recognition module 214 can use various models to recognize objects appearing in the image data associated with the 3D model, including but not limited to: extracted features and reinforcement learning algorithms, bag-of-words models with features such as Speeded Up Robust Features (SURF) and Maximum Stable Extremal Regions (MSER), gradient-based and derivative-based matching methods, Viola-Jones algorithm, template matching, deep learning, image segmentation, and blob (binary large object) analysis.

[0080] In some embodiments, the object recognition module 214 can determine detailed features associated with the identified objects to improve the accuracy of identifying potential tags associated with each object. The detailed features can include any potential features about the object that the object recognition module 214 can perceive. For example, the detailed features can relate to one or more colors of the object, the type of the object, the size of the object, the material of the object, the model or brand associated with the object, the identity associated with the object, etc. For example, in addition to detecting an appliance included in the kitchen of the 3D model, the object recognition module 214 can determine that the appliance is a microwave oven, determine the brand of the microwave oven, the model of the microwave oven, and the size of the microwave oven.

[0081] In some embodiments, the object recognition module 214 can use contextual information received in association with the capture process to help identify the objects included in the 3D model. For example, in some implementations, during the capture process, the user can provide spoken notes or annotations about the captured environment. For example, when the user scans the living room with a 3D capture device, the user can provide spoken symbols about the objects that the user sees in person and / or in the captured 2D / 3D image data. For example, the user can state that the living room has hardwood floors, the type of wood of the floor, the company from which the floor was purchased, and any other possible descriptive information that the user may know, which can help identify the floor and / or the label data associated with the floor when the floor is presented in the 3D model of the living room. These spoken symbols can be associated with a specific image data capture (e.g., a 2D image) and position information associated with the image data capture, including position information for pixels or points respectively contained in the 2D image. In addition, the object recognition module 214 can use sensor measurement information (e.g., temperature, pressure, light, etc.) associated with the captured 2D or 3D image data of the environment to help identify the objects included in the 2D or 3D data. Thus, in some embodiments, object recognition module 214 may use contextual clues previously associated with image data of the 3D model to recognize and characterize objects in the 3D model.

[0082] In addition, the target recognition module 214 can use contextual annotation information received during the capture process, rather than attempting to identify and characterize all potential markable targets included in the 3D model, which contextual annotation information indicates which targets are specifically identified and characterized for the purpose of marking. For example, the contextual annotation information can include verbal annotations provided by the user during the capture process and image-based contextual information. For example, a user can take a close-up photo or video of a target and / or a logo on the target to assist in automatic target recognition of the target and indicate a label that is desired to be associated with the target. In other embodiments, a user can provide input to select a point or a group of points included in 2D or 3D image data associated with a 3D model to which the user wishes to apply a label (e.g., including captured image data before alignment and a representation of a 3D model generated after generating the 3D model). The target recognition module 214 can also be configured to analyze visual and structural features associated with the point or a group of points to determine and characterize the target associated therewith.

[0083] In various embodiments, the labeling module 216 defines labels for one or more targets and can associate labels with specific points in the three-dimensional model. In one or more implementations, the labeling module 216 can add signs or mark specific points of interest, areas, targets, target instances, etc. The labels can be depicted in the visualization by the visualization module 218 (discussed herein). The labels can include or be text, images, videos, audio and / or hyperlinks. In various embodiments, the labeling module 216 identifies (e.g., through labels and / or signs) targets, paths, or points of interest in the 3D model visualization. The labeling module 216 can use the labels from the target recognition module 214 to identify the targets.

[0084] In some embodiments, a user interacting with the GUI and / or the visualization of the 3D model may provide tags to be associated with objects, points, etc. of the 3D model. For example, after selecting or identifying a point, region, or object of interest, a user may create, upload, attach, embed, link, select, or otherwise provide input to the tag module 216, including tags associated with the point, region, or object of interest. In some embodiments, tags are used to describe or provide additional information (e.g., semantic information, sentiment information, historical information, incidental information, etc.) of the point, region, or object associated with the tag.

[0085] In some embodiments, the tag module 216 may associate additional metadata with the tag, such as information about the author or creator of the tag, a timestamp associated with the creation of the tag, a three-dimensional model associated with the tag, the location of the tag relative to the 3D model, the location of the tag relative to the physical environment (e.g., floor), the target type or target category, etc. In general, tags can be applied to or associated with points, areas, or targets in a 3D model. For example, in some implementations, tags can be associated with discrete 3D points associated with a 3D model. Tags can also be associated with a set of 2D or 3D points on or within a 3D model. In another example, a tag can be associated with a face / surface or a set of faces / surfaces included in the 3D model. In another example, a tag can be associated with an edge of an edge group located on a structural surface of the 3D model. In another example, a tag can be associated with a volume or a set of volumes that are enclosed or contain one or more structural surfaces in a three-dimensional model. Tags can also be associated with an area or volume of free space in a model. Tags can also be applied to or associated with a visual image or a subset of images that are part of a 3D model. In another aspect, tags can be associated with a specific viewpoint relative to a 3D model. For example, when applied to a building, a tag can be associated with a specific viewpoint of the building, a room in a building, a wall in a building, a fixture in a building, furniture in a building, decoration in a building, etc. The size, shape, and appearance of the tag icon may vary. In some embodiments, the size, shape, and appearance of the tag icon of the tag can be cropped based on the preferences of the author or system of the applied tag. For example, the author can choose a specific symbol to represent the tag, the size of the symbol, the size of the symbol, the color of the symbol, the opacity or skew of the symbol, etc. In other implementations, different types of tag icons and / or the appearance of the tag icon can be used to distinguish the characteristics of the tag represented by the tag icon. For example, different types or colors of tag icons can be used to identify different authors or entities that apply tags. In another example, different types or colors of tag icons can be used to specify the type of tag data or tag metadata, such as whether the tag provides text data, image data, audio data, video data, hyperlink data, etc. For example, a music symbol can be used to represent tag data including music, or an image of a film tape or a camera can be used to represent tag data including video. In another example, different types or colors of tag icons can be used to specify a class or category associated with a tag. In another example, different types or colors of tag icons may be used to indicate a timestamp associated with a tag.

[0086] The specific tags and associated tag icons included in the representation of the 3D model presented to the user may also vary based on various factors, including factors related to the preferences of the author or entity applying the tags, and factors related to the user viewing / consuming the 3D model and viewing context. For example, various points, areas, or targets in the model may be associated with different tags. In addition, a single point, area, or target in the 3D model may be associated with multiple tags. The specific set or subset of tags and associated tag icons associated with the 3D model presented to the end user may vary based on the characteristics of the end user (e.g., preferences, demographics, etc.), the client device presenting the 3D model, and the viewing context (e.g., location, time of day, day of year, mobile state, etc.). In some embodiments, an entity that applies tags or otherwise has authority over how to present tags and associated tag icons to end users may specify when to display one or more tags and associated tag icons to the end user. The tag module 216 may integrate and / or display tags and associated tag icons in the representation of the 3D model accordingly. For example, an entity may customize the specific tags and / or tag icons displayed to an end user based on characteristics of the end user (e.g., preferences, profile, demographics, etc.), characteristics of the client device (e.g., type and size of display), viewing context (e.g., including location, time of day, day of year, mobile status, etc.), etc.

[0087] In some embodiments, interaction with a particular tag icon can trigger a response associated with one or more related tags. For example, in some implementations, tags can be grouped according to various factors. For example, tags can be grouped according to association with the same room, the same type of target, the same date, the same manufacturer, the same price factor, the same author, the same decorator, and the like. According to these embodiments, selection of or interaction with one of the tag icons for tags included in a defined group can cause any number of other tags in the group to assume a particular state, such as being activated, highlighted, hidden, changed in size, shape, or position, and the like. For example, in one implementation, selecting a tag icon for a tag included in a tag group may result in displaying only the tag icons for the tags in the group. Other tag icons in the model that are not associated with the group may be removed from the display or displayed in a lighter or more transparent color. In another example, selecting a tag icon for a tag included in a tag group may result in highlighting only the tag icons for the tags in the group. In another example, selecting a tag icon for a tag included in a tag group may result in tags for all tags in the group being displayed in a side panel. In one example, a user can Figure 4The search query is submitted in the search field 402 of to find the sockets in the three-dimensional model of the house 110. The object detection and positioning system 112 can receive the search query. The request searches the object data store 220 for object type entries and object instance entries corresponding to the search query. Figure 4 In the example of , the target data store 220 can receive a search query for a socket. The target data store 220 is searched to find target instance entries that have been classified or recognized as sockets. In some embodiments, each target instance entry is associated with a specific socket found in the house 110. Each target instance entry may include identifiable information about the target, such as the target's category and unique target identifier and attributes. In addition, the target data store 220 can perform a search for target type entries corresponding to the socket. The target type entry may include an incremental count of the number of sockets in the 3D model of the physical environment.

[0088] The visualization module 218 may implement visualization or tour of the 3D model. The 3D model may be generated based on two-dimensional images and / or depth data provided by one or more image capture devices 106. In various embodiments, the visualization module 218 provides the visualization or tour to one or more other digital devices (e.g., by downloading or streaming code) for user interaction (e.g., to tour the 3D model).

[0089] In some embodiments, the visualization module 218 provides interaction through visualization through an API or other means. In one example, a user can provide a query or interact with a GUI element provided by the visualization module 218 to request information about one or more targets identified by the target recognition module 214. The request can be for a variety of different target types (e.g., the number of electrical targets on a floor or in a residence), the location of one or more different target instances in a 3D model or physical environment, etc. The visualization module 218 can retrieve information previously stored by the target recognition module 214 and / or the labeling module 216 and provide a response. In some embodiments, the response is in text form (e.g., a list or number) and / or a visualization can be used to indicate the location of the target instance in response to the request.

[0090] The target data store 220 can be any one and / or multiple structures suitable for storing target instance entries or target type entries. For example, the target data store 220 can be an active database, a relational database, a self-referencing database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, etc. The target data store 220 has information about the location, target type, or quantity of the targets identified by the target recognition module 214.

[0091] In some embodiments, the target data store 220 stores information about the target in a target instance entry. The target instance entry may contain or indicate a specific instance of a target type. A target type entry may be associated with a specific target type. In some embodiments, the target data store 220 may store a target type number indicating the total number of targets of the target type on a floor of a 3D model, a portion of a 3D model, and / or the entire 3D model. For example, when the target recognition module 214 recognizes an instance of a dining room chair, the target recognition module 214 may send a request to the target data store 220 to create a target instance entry associated with the specific dining room chair. In addition, the target recognition module 214 may increase the incremental count of the target type entry associated with the dining room chair.

[0092] Each target instance entry can be associated with a target identified by target recognition module 214. Target instance entries can include recognizable information about the target, such as a category and a unique target identifier. Target instance entries can also include attributes of the target, such as 3D coordinates (points in 3D Euclidean space), physical properties of the target, such as the estimated size and color of the target, and the brand of the target (if applicable). In some embodiments, target data storage 220 includes attributes of a label or icon representing a target in a three-dimensional model, such as the size, shape, or positioning of an icon representing the target.

[0093] Image data store 222 may be any one and / or more structures suitable for digital images, including two-dimensional panoramic images, three-dimensional panoramic images, 3D models, etc. As with target data store 220, image data store 222 may include or be an active database, a relational database, a self-referencing database, a table, a matrix, an array, a flat file, a document-oriented storage system, a non-relational No-SQL system, an FTS management system such as Lucene / Solar, etc. In some embodiments, target data store 220 stores depth data generated by depth data module 210. In various embodiments, target data store 220 stores attributes associated with an image capture device or characteristics associated with each of a plurality of image captures or depth captures used to determine a three-dimensional panoramic image.

[0094] Figure 3 A flow chart of an object detection process 300 is depicted in accordance with some embodiments. In various embodiments, the communication module 204 receives a plurality of two-dimensional images and / or a three-dimensional model of the physical environment. In some embodiments, the communication module 204 receives any number of two-dimensional images from the image capture device 106. In some embodiments, the communication module 204 may receive depth information from the image capture device 106 or another device.

[0095] (If a 3D model is received) The 3D model may be created by a third party software application (not shown). In some embodiments, the communication module 204 receives the two-dimensional image and / or the three-dimensional model from other sources, such as the model data store 116. In one example, Figures 7 to 10 Views 700 , 800 , 900 , and 1000 , respectively, depict examples of a set of two-dimensional images of a physical environment from different viewpoints.

[0096] In some embodiments, the alignment module 208 can align or "stitch" the various 2D images captured by the 2D / 3D panoramic capture device (e.g., the image capture device 106) to generate multiple two-dimensional images. For example, the alignment module 208 can also use the GPS coordinates of 106 during the capture of each two-dimensional image to determine the alignment of one or more two-dimensional images.

[0097] The alignment module 208 may also use depth information about the relative positions of the various visual features included in the various 2D images to further align the various 2D images with each other in association with generating a 2D panoramic image. The alignment module 208 may also perform blending and / or graphic cutting at the edges to remove seams. In various embodiments, the alignment module 208 may utilize characteristics or attributes of the two-dimensional images, such as color or light intensity, to align or stitch the imaging together.

[0098] In some embodiments, the object detection and localization system 112 may generate a 3D model based on any number of 2D images and / or depth data.

[0099] In step 302, the object recognition module 214 can recognize the object. In various embodiments, the object recognition module 214 recognizes the object based on the 2D image and the associated position information, depth data and / or mesh data of the image capture device. These different processes are discussed herein. The object recognition module 214 can recognize the object captured in the 2D image and / or the 3D model.

[0100] In step 304, the labeling module 216 may classify and / or label the objects in the 3D model as object instances and / or by object type. In various embodiments, the object recognition module 214 and / or the labeling module 216 collect information about the location of the objects recognized by the object recognition module 214 in the 3D model, the number of object instances of one or more object types, and the like.

[0101] In step 306 , information collected or generated by the object recognition module 214 and / or the tag module 216 may be stored in the object data store 220 for later retrieval to provide information relevant to an information request (eg, from a user).

[0102] In step 308, the visualization module 218 provides the 3D model visualization to a digital device or enables a user to view and / or interact with the 3D model (e.g., as a tour of a virtual environment). The visualization can be interactive, where the visualization includes a graphical user interface (GUI) that enables a user to request information (e.g., by querying fields) or interactive elements of the GUI.

[0103] In step 310, the visualization module 218 receives an information request from a user (eg, or another digital device). The information request may be for the number or location of a particular object type within the 3D model.

[0104] In step 312 , the visualization module 218 retrieves information (eg, from the target data store 220 and / or the image data store 222 ) in response to the request.

[0105] In step 314 , the visualization module 218 provides a visualization of the response and / or the object in the 3D model in response to the request.

[0106] Figure 4 is an example user interface 400 associated with a visualization. Figure 4 In the example, the user can interact with area 410 to view and navigate the portion of the building. In this example, the user requests more information about the location of the socket in the 3D model by providing a query in the query field of the visualization (e.g., provided by the visualization module 218).

[0107] In response, the visualization module 218 can retrieve information about objects of the requested object type ("sockets"), can provide a list 420 of identified sockets, and / or provide a view (eg, a dollhouse view) indicating the sockets in the image.

[0108] Although in Figure 4 Although a socket is used in the example of FIG, it will be appreciated that a user may search for any object that has been identified and associated with stored information by the object detection and location system 112 .

[0109] By providing information about the 3D model and the associated physical structure, the user may be able to do more than just "tour" the 3D model. Instead, the user may interact with the visualization, identifying key parts of the model of interest (e.g., number of bathrooms, positioning of sinks, etc.) to ensure that the premises is suitable for the user's intended purposes.

[0110] By enabling automatic target detection and target information collection, the target detection and localization system 112 can provide interactive visualization and previously unavailable analysis capabilities to users in a manner that scales with any number of 3D models for any number of users or other digital devices. In addition, by identifying targets and collecting information as described herein, errors are reduced, information is more easily obtained, and computing resources are greatly saved compared to manual data collection methods.

[0111] In some embodiments, a user may interact with list 420 that provides multiple instances of targets within a given target type (ie, target category).

[0112] Back to Figure 4 In the example of FIG. 4 , list 420 lists identified outlets located in the 3D model. A user can interact with a specific outlet, such as entry 422. In response, user interface 400 can provide the location of the specific outlet in the floor plan of the building.

[0113] Figure 5 A flow chart of an object detection process 500 is depicted in accordance with some embodiments. As described herein, in various embodiments, the communication module 204 receives a plurality of two-dimensional images and / or a three-dimensional model of a physical environment. In some embodiments, the communication module 204 receives any number of two-dimensional images from the image capture device 106. In some embodiments, the communication module 204 may receive depth information from the image capture device 106 or another device. In some embodiments, the object detection and localization system 112 may generate a 3D model based on any number of 2D images and / or depth data.

[0114] In step 502, the object recognition module 214 can recognize the object. In various embodiments, the object recognition module 214 recognizes the object based on the 2D image and the associated position information, depth data and / or mesh data of the image capture device. These different processes are discussed herein. The object recognition module 214 can recognize the object captured in the 2D image and / or the 3D model.

[0115] In step 504, the labeling module 216 may classify and / or label the objects in the 3D model as object instances and / or by object type. In various embodiments, the object recognition module 214 and / or the labeling module 216 collect information about the location of the objects recognized by the object recognition module 214 in the 3D model, the number of object instances of one or more object types, and the like.

[0116] In step 506, the object recognition module 214 may increment an object type count (e.g., an object type number) for each object instance of a particular object type. For example, the object recognition module 214 may increment a socket count (e.g., a "socket" object type) for each socket instance that the object recognition module 214 recognizes for a particular 3D model or physical environment.

[0117] In step 508, target recognition module 214 may collect information about the location of any number of target instances recognized by target instance module 214. For example, target recognition module 214 may recognize the location of the target based on the location of an image capture device that captured an image depicting the target, based on depth data, and / or based on a mesh of a 3D model. The location information may indicate the floor or portion of the 3D model where the target instance is located. In some embodiments, the location information indicates the location of a particular target instance in the 3D model.

[0118] Object recognition module 214 may associate object type counts and / or location information with the object instances and store the information in depth data module 210 to enable visualization module 218 to respond to queries regarding the object(s) of the 3D model.

[0119] In step 510, the visualization module 218 provides the 3D model visualization to a digital device or enables a user to view and / or interact with the 3D model (e.g., as a tour of a virtual environment). The visualization can be interactive, where the visualization includes a graphical user interface (GUI) that enables a user to request information (e.g., by querying fields) or interactive elements of the GUI.

[0120] In step 512, the visualization module 218 receives a query from a user (eg, or another digital device) via a visualization API or field. In this example, the user can provide a query about the number and location of a specific target type (eg, fire extinguishers).

[0121] In step 514, the visualization module 218 retrieves information (e.g., from the target data store 220 and / or the image data store 222) in response to the request. In this example, the visualization module 218 identifies the requested target type (i.e., fire extinguishers) and retrieves a count of the target type (e.g., the number of fire extinguishers) for the 3D model. It should be understood that a user may request the number of fire extinguishers per floor, the location of the 3D model, etc. In this example, the visualization module 218 may limit the information request or filter the information request so that only the number of target instances that meet the query criteria are retrieved. In one example, the visualization module 218 may identify target instances that meet the target type identified in the query, which target instances are located at or within a specified location of the model, determine the total number of target instances that meet the criteria, and return the number of target instances that meet the criteria.

[0122] The query may also request the location of one or more fire extinguishers in all or part of the 3D model.The visualization module 218 may retrieve object information indicating the location of the objects of interest (ie, the location of the object instances associated with the object type of "fire extinguisher").

[0123] In step 516, the visualization module 218 may provide the retrieved information (e.g., the number of target instances of the desired target type) and the locations of the target instances that meet the criteria. In some embodiments, the visualization module 218 may provide a list of the locations of the targets of interest and / or an indication of the locations of the targets of interest in the 3D model (e.g., in a dollhouse view, the user may see a portion of the 3D model, one or more floors of the 3D model, or a perspective view of the entire 3D model).

[0124] Figure 6 1 is a flow chart for identifying instances of the same or different objects in two-dimensional images based on the position of the image capture device that captured the images in some embodiments. Although there are many ways to determine whether objects in multiple images are the same / same object (e.g., using only depth data and the position of the depth data in a 3D model), Figure 6 The process described in can be scaled up as the number of images and the number of objects in the images increase.

[0125] If the same object is recognized in multiple images, the description of the object in different images (e.g., different views) can be used in the 3D model. For example, a user can control the viewpoint in a tour of the 3D model. As the viewpoint changes, the view of the same object (e.g., the same chair) may change depending on the viewpoint. An image of a chair including different depictions in a 3D model can be generated based in part on different viewpoint depictions of the object in different images. By using the position and orientation of the camera to identify the same object in different images, pixels and other information can be presented in the 3D model.

[0126] Furthermore, by associating an object with the position and orientation of the image capture device, objects may be distinguished from one another. For example, two objects captured in different images of the same physical environment may be similar to one another (i.e., they may be the same object type). However, based on the position of the image capture device, it may be determined that the two objects are different from one another (e.g., the objects depicted are both chairs, but on different floors), thereby allowing the similar objects to be distinguished, allowing a count of the object type to be incremented, and storing metadata associated with the different objects (e.g., positioning relative to the 3D model and / or physical environment).

[0127] In step 602, the communication module 204 receives a plurality of two-dimensional images of a physical environment (eg, the interior of a facility, building, or residence). Each image may be associated with a position and / or orientation of an image capture device that captured the image.

[0128] In step 604, the object recognition module 214 may identify and / or recognize a first object depicted in the particular image. The depicted first object may be from a particular view. The depicted view of the object in the particular image may be at a particular angle relative to the first object, may be partially obscured in the image, etc. The first image may be associated with a position and / or orientation of the image capture device 106 that captured the image (i.e., when the image was captured by the image capture device 106).

[0129] In step 606, the object recognition module 214 may identify and / or recognize a second object depicted in the particular image. The depicted second object may be from a particular view (e.g., a different view than the view depicted in the first image). The depicted view of the second object in the particular image may be at a particular angle relative to the second object, may be partially obscured in the image, etc. The second image may be associated with a position and / or orientation of the image capture device 106 that captured the image (i.e., when the image capture device 106 captured the second image). The position of the image capture device 106 may be different from the position of the image capture device 106 when the first image was captured.

[0130] The location of the first target and the second target can be determined in a variety of ways. In some embodiments, the location of the first target and the location of the second target can be determined at least in part using depth data from a depth data sensor (e.g., LiDAR) (if available). In various embodiments, the image capture device 106 includes a camera and LiDAR and is configured to acquire images and depth data of the physical environment (e.g., using measurements from a LiDAR sensor). The targets depicted in the images can be associated with the depth data to identify the location and / or if two or more targets in two or more images depict the same target (e.g., by determining that the targets in each image coincide with the same location defined by the depth data).

[0131] Understandably, many objects may appear similar, but they are different instances of similar objects (e.g., two kitchen chairs that look very similar based on the image, or two electrical sockets that appear to have different designs). Likewise, some objects may appear identical (e.g., the same model of fire extinguisher on each floor), but are distinct (and distinguishable based on localization).

[0132] It should be understood that any number of image capture devices 106 may capture any number of images (e.g., multiple images may be captured by repositioning the same image capture device 106 or using any number of image capture devices 106 to capture any number of images from similar or different locations in the physical environment).

[0133] The object recognition module 214 can recognize objects in the image in any number of ways. For example, the object recognition module 214 can utilize a trained neural network (e.g., R-Mask CNN) to recognize any number of objects (e.g., using instance segmentation). In some embodiments, the neural network generates a confidence score associated with the confidence that the object was recognized and / or classified. In various embodiments, the object recognition module 214 can apply a confidence threshold to delete object recognition(s) and / or classification if the confidence score is insufficient relative to the confidence threshold.

[0134] In step 608, the object recognition module 214 maps pixels (e.g., a portion of the image associated with the first object) to locations in the 3D model and / or physical environment based at least in part on the position and / or orientation of the image capture device 106 when the first image was captured.

[0135] In step 610, object recognition module 214 determines whether the second object in the second image is a second view of the same object in the first image (e.g., the second object is a different view of the first object). In some embodiments, object recognition module 214 maps pixels (e.g., a portion of the image associated with the second object) to locations in the 3D model and / or the physical environment based at least in part on the position and / or orientation of image capture device 106 when the second image was captured. If the mappings are consistent, then if both the first object and the second object are the same object type (e.g., they are both classified as part of the same object type), object recognition module 214 can conclude that the first and second objects in the different images depict the same object in the physical environment.

[0136] In various embodiments, rather than mapping pixels or images of the second image, the object recognition module 214 may utilize the position and / or orientation of the image capture device 106 when the second image was captured to determine whether the first object and the second object coincide in the 3D model and / or physical environment.

[0137] If the target recognition module 214 determines that the targets are the same, then in step 612, the 3D model can utilize pixels and / or image portions of different views of the same target within the 3D model (e.g., within a visualization of the 3D model) to improve the accuracy and realism of the target depicted in the 3D model. In various embodiments, the target recognition module 214 can collect target information about a single target depicted in two images. For example, the target recognition module 214 can store metadata associated with the target, such as the target type, the number of increments of the particular target type, and / or the location of the target. The target information (e.g., associated with the target) can be stored in the target data store 220.

[0138] If the target recognition module 214 determines that the first and second targets are different (e.g., they are in different locations in the physical environment), then in step 614, the target recognition module 214 can collect separate target information for each target. In various embodiments, the target recognition module 214 can collect target information about two targets (e.g., the first and second targets) depicted in different images. For example, the target recognition module 214 can store metadata associated with the targets, such as the target type, the number of each target of the particular target type, and / or the location of the target. The target information (e.g., associated with each target) can be stored in the target data store 220.

[0139] In various embodiments, pixels within different views of the same object depicted in different images can help visualize the object within the 3D model. In addition, by identifying objects within images of the physical space, metadata associated with the different objects (e.g., labels, locations, etc.) can be stored to enable querying of the 3D model and / or the physical space.

[0140] Information associated with different objects (e.g., location, number of objects of the same type, etc.) may be collected and stored. This may enable an interactive system where a user can query a 3D model for the number of specific objects (e.g., "What are the number of electrical outlets?") and / or the location of specific objects (e.g., "Where is the fire extinguisher?"). The system can retrieve answers from the collected information associated with the location and number of different objects / object types to provide textual and / or visual answers.

[0141] Figure 7-10 Depicts different images of an example taken at different locations within a physical environment. Figure 7-10 In the example above, take an image of a warehouse or retail store. Figure 7 A view 700 is depicted, which includes, for example, also with Figure 8-Figure 10 The other images in the image appear in different angles and viewpoints. Figure 7 , target 702 is partially visible.

[0142] Figure 8 Depicted with Figure 7 800 within the same physical environment depicted in , but may or may not include the same target.

[0143] Fig. 9 Depicted with Figure 7 , but includes object 902. Object 902 may be object 702. In various embodiments, object recognition module 214 may use object recognition (e.g., using instance segmentation) to identify the objects in both images, and may utilize orientation and / or position information of image capture device 106 that captured the images to determine whether the same object appears in both images.

[0144] Fig.10 Depicted with Figure 7-Figure 9 FIG. 1000 is a diagram showing the same physical environment. Fig.10 The target 1002 identified in the two images may be the target 702 and / or the target 902. In one example, the target recognition module 214 determines whether the targets are the same target type, and if they are the same target type, the target recognition module 214 may determine whether the targets are the same based on the position and orientation of the image capture device(s) 106 that captured the image. In various embodiments, the target recognition module 214 may use target recognition (e.g., using instance segmentation) to identify the targets in the two images, and may utilize the orientation and / or position information of the image capture device 106 that captured the images to determine whether the same target appears in any number of images.

[0145] Fig.11 Depicted Figure 7-10 1102, wherein the positions of the image capture device and the positions of the two-dimensional detected objects are shown. In this example, different capture positions 1110, 1112, 1114, etc. may be positions where images including object 1102 were taken. Each capture position 1110, 1112, and 1114 may be a position of image capture device 106 that captured the image. In various embodiments, the orientation of image capture device 106 when the image was captured is also associated with the image.

[0146] For example, target 1102 may be the same target as targets 702, 902, and 1002. In this example, each different capture location may be associated with Figure 7 , Fig. 9 and Fig.10Based on the position and orientation of image capture device 106 when the images were captured, the objects may be triangulated to confirm or determine (e.g., with a reasonable probability) that similar objects in different images may be the same object (or may be different objects).

[0147] Fig.12 A flow chart for collecting information and providing interaction with a 3D model in some embodiments is depicted. In step 1202, the object recognition module 214 classifies or identifies objects in one or more images. The object recognition module 214 can classify the object and / or save the object information (e.g., the location of the object in the 3D model or physical environment) in the image data storage 222.

[0148] In some embodiments, object recognition module 214 utilizes a neural network to recognize objects in a two-dimensional image (eg, of a physical environment captured by image capture device 106). It should be understood that one or more two-dimensional images may include one or more recognized objects.

[0149] As described herein, the same recognized object may be recognized in different two-dimensional images from different viewpoints. In some embodiments, object recognition module 214 may generate a bounding box that depicts the boundaries or outer limits of a particular region of interest (eg, that may contain an object to be recognized or identified).

[0150] Fig.14 An overview of process 1400 for an implementation of a target detection and localization system is depicted. Element 1402 is a 3D model and is both an output and an input in process 1400. Element 1404 of process 1400 includes neural network 1, neural network 2, and neural network 3 (which may be differently trained neural networks) and may perform 2D processing, where different neural networks are trained to recognize different targets. Element 1406 may associate pixels of the identified target with a three-dimensional surface of the 3D model. Element 1408 may merge detections, and element 1408 may perform classification of target types based on detections and semantics. Process 1400 may label one or more identified targets with labels located within a visualization of the 3D model.

[0151] In some embodiments, the neural network can perform object recognition (e.g., using instance segmentation) to provide a probability that a particular object in an image belongs to a particular category (e.g., object type).

[0152] For each target identified by the neural network, the target recognition module 214 can identify each pixel in the subset of two-dimensional target detections that is associated with the particular target. These identified pixels can be referred to as target pixels. In addition to identifying target pixels of one or more targets, the target recognition module 214 can also associate the target pixels with panels or surface elements of a three-dimensional model of the physical environment.

[0153] For each target, target recognition module 214 may obtain collected target data (e.g., the position and / or orientation of image capture device 106 that captured the target image, an inference of the position of the target relative to one or more positions and / or orientations, etc.) Determining the proximity of two-dimensional target detection subsets to each other may more clearly indicate the probability that two or more identified targets in different images are the same target.

[0154] In some embodiments, the target recognition module 214 can provide a cumulative confidence of a specific target recognized in a set of three-dimensional target detections based on the three-dimensional target probability. The cumulative confidence can be aggregated, averaged, weighted averaged, etc. In various embodiments, the aggregate confidence can be calculated using the following equation:

[0155]

[0156] where c i is the confidence of the i-th target detection report in the set of 3D target detections for a specific target. Figures 17A-17C Depicts the same chair in different images and the probability score of the object being recognized. Figures 17A to 17C In the example of , it provides a set of 3D object detections 1712, 1722, and 1732, each with a 3D object probability of 0.86, 0.62, and 0.94, respectively. The aggregate confidence can be determined as follows:

[0157] Confidence combined =1-(1-0.86)(1-0.64)(1-0.94)=0.997 Once the target recognition module 214 determines the cumulative confidence of a particular target, the target recognition module 214 can send a request to the target data store 220 to update or create a target instance entry for the particular target (e.g., the targets are all the same target instance, the targets are of the "chair" target type, and / or the target instance is associated with a location in the 3D model and / or physical environment). In addition, the target recognition module 214 can send a request to the label module 216 to define a label and a specific point in the three-dimensional model to associate the label.

[0158] Fig. 20A user interface 2000 is depicted that includes a label 2010 that provides a category for a particular target and a label icon 2020. A user can interact with the user interface 2000 in an area of ​​1720 to receive more information about the particular target. It should be understood that the scan and confidence values ​​may or may not be provided to the user in the user interface 2000.

[0159] In some embodiments, three-dimensional detections of the same object from two different views may be represented as two different sets of points (e.g., surfaces) in a 3D model (e.g., detection surfaces that may be described in the same coordinate system given a priori knowledge of the relative positions of the image capture devices viewing and capturing the images). As a result, object detection and localization system 112 may "merge" the detection surfaces from the different views (i.e., associate the union of the two sets of points with the same physical object) for depiction in a visualization of the 3D model.

[0160] One example of a method to accomplish this may be to assume that a surface of an object visible from one view (e.g., the front top of a table) may be close to or overlap another surface of the same object visible from a different view (e.g., the back top of the same table). Object recognition module 214 may automatically determine that detections of these two surfaces representing the same object may utilize these assumptions to make its determination. For example, object recognition module 214 may consider the distance between the centroids of the points of the two detection surfaces and the fact that they are close to each other; or, that each point in one of the detection surfaces has a sufficient number of points from the other detection surface that is located close enough to it.

[0161] Fig.14 An overview 1400 of an implementation of an object detection and localization system is depicted. Element 1402 corresponds to step 302 of object detection process 300. Element 1404 of overview 1400, including neural network 1, neural network 2, and neural network 3 of overview 1400, and a plurality of two-dimensional detections corresponding to their respective neural networks, corresponds to step 306, in which different neural networks are trained to recognize different objects. Element 1406, in which a pixel is associated with each two-dimensional object detection having a three-dimensional surface of a model, may correspond to step 310. Element 1408 may correspond to step 312, in which a set of three-dimensional object detections are identified. Return to Fig.12At step 1204, if there are multiple different targets (e.g., located at different locations relative to the physical environment and / or the 3D model) of the same target type (e.g., identified as the same category or the same target type), then the target type number for each different target of the target type can be incremented to track the number of target types. The same target type may have different target type numbers. For example, there may be an overall target type number to track the total number of targets of the same target type in the 3D model. A portion of the 3D model may also have a target type number (e.g., to track the count of different targets of the same type in a room, floor, etc.).

[0162] In step 1208, the visualization module 218 may provide an interface to allow the user to interact with the 3D model. The visualization module 218 may receive a query from the user through the interface, or may receive the query through different means (e.g., through an API). The query may request information about one or more objects. For example, the query may request the number of a particular object type in the 3D model, the location of objects of the same object type, etc.

[0163] In step 1210, the visualization module 218 can retrieve from the target data store 220 a target type number associated with the desired target type identified in the query, as well as location information associated with the requested target type identified in the query (e.g., the number of electrical outlets on the floor in the physical environment and the location of the electrical outlets).

[0164] In step 1212, the visualization module 218 provides a response to the user. For example, the visualization module 218 can provide a target type number (e.g., the number of electrical outlets) and the location and / or position of target instances of the same target type identified in the query. In various embodiments, the visualization module 218 can provide information via text (e.g., a list), instructions within a 3D model, graphics, audio, animation, etc.

[0165] Fig.13 Depicted is a flow chart of a target detection process 1300 to a target detection process according to some embodiments. In step 1302, the communication module 204 receives a previously generated three-dimensional model of a physical environment. The three-dimensional model can be created by a third-party software application (not shown). The three-dimensional model can be generated based on a two-dimensional image and depth data of the physical environment. In some embodiments, the three-dimensional model does not include a grid.

[0166] In step 1304, the mesh construction module 212 may generate a mesh of the physical environment. The mesh construction module 212 may generate the mesh based on the depth data received from the physical environment. In some embodiments, the mesh construction module 312 generates a triangular mesh, which may be Fig.15A An example can be found in . Fig.15A An image depicting a physical environment covered with a triangle mesh. Fig.15A The pixels within the grid element 1502 represent an area of ​​the environment on a common physical plane.

[0167] In step 1306, the object recognition module 214 can recognize objects within the three-dimensional model received in step 1302. In some embodiments, the machine learning algorithm is trained to recognize three-dimensional objects and / or two-dimensional objects. The object recognition module 214 using the trained machine learning algorithm (e.g., a neural network and / or any other ML method) can detect and recognize three-dimensional objects based on the three-dimensional model and the mesh received and / or generated in step 1304.

[0168] The object recognition module 214 can either recognize the objects in the grid (e.g., in step 1306) or classify the objects using a machine learning algorithm (e.g., a neural network discussed herein) in step 1308.

[0169] As described in this article, grids can help define positioning and distances within a 3D model. For example, Fig.15A The view 1500 of FIG. 1 depicts a view of a three-dimensional model including mesh elements 1502. The object recognition module 214 can recognize two pieces of furniture, sofas 1504 and 1506. Each sofa 1504 can be depicted on the view 1500 as a series of triangles including mesh elements 1502 that are colored in different colors.

[0170] Object recognition module 214 recognizes pixels of the three-dimensional model associated with the detected three-dimensional object. Once object recognition module 214 determines that the three-dimensional views in the set of previously recognized objects are three-dimensional views of the same object, object recognition module 214 can refine the edges of the recognized objects. For example, object recognition module 214 can define a mesh surface of sofa 1504, including mesh elements 1508. The three-dimensional surface corresponding to mesh elements 1508 in physical space is not part of sofa 1504. As object recognition module 214 views sofa 1504 from different views of the three-dimensional model, the edges or boundaries of sofa 1504 can be further refined.

[0171] Fig. 15B The view 1520 includes a representation 1510, namely Fig.15A A finer edge of sofa 1504 is identified in FIG. 1 , which does not include mesh element 1508 , such as Fig.15A shown.

[0172] Fig.16A and Fig. 16BThe views 1600 and 1620 of FIG. 1 and 2 provide another example of an object recognized by the object detection and positioning system 112. The object recognition module 214 can recognize a shape representing the recognized object, namely, a bed 1602, which includes a mesh element 1604. In this example, Fig.16A The representation 1606 of the bed 1602 identified in FIG. 1 covers more than the actual physical space occupied by that particular piece of furniture. Fig. 16B As shown, region 1608 of representation 1606 is not part of the bed, but object recognition module 214 recognizes the region as part of the same recognized object. Fewer views of the three-dimensional model of the bedroom may cause the edges or boundaries of the recognized object to be misaligned with reality. In some embodiments, the edges of the recognized object can be manually corrected by a user of the object detection and positioning system 112.

[0173] The object recognition module 214 may provide a cumulative confidence of a particular object recognized in the set of three-dimensional object detections based on the three-dimensional object probabilities.

[0174] In step 1310, the object recognition module 214 may send a request to the object data store 220 to update or create an object instance entry for a particular object. In addition, the object recognition module 214 may send a request to the tag module 216 to define a tag and a particular point in the three-dimensional model to associate the tag with.

[0175] Fig.18 1800 according to some embodiments. In optional step 1802, image capture device 106B may capture a digital image of a structure, such as house 110. In step 1804, communication module 204 may receive the captured two-dimensional image of house 110. An example of a two-dimensional image captured by image capture device 106B may be Fig.19 A two-dimensional panoramic image 1900.

[0176] During this process, the communication module 204 receives a two-dimensional image of the physical environment of the interior and / or exterior of the house 110. Before identifying an object depicted in the digital image of the house 110, the object detection and localization system 112 can estimate depth data associated with the two-dimensional image. In step 1806, the depth data module 210 can determine the depth data associated with the two-dimensional image received by the communication module 204. The depth data module 210 may include a machine learning algorithm that estimates the depth data of the two-dimensional image based on features such as color or light intensity to determine the distance between a spatial point depicted in a particular pixel of the image and the image capture device that captured the image. The estimated depth data can be used to generate a three-dimensional model of the physical environment. Fig.19An example of a depth map of a physical environment can be seen in Depth map 1910 maps each pixel to a measure of the distance from the optical center of the image capture device to the nearest physical surface traversed by a ray traced from the camera's focal point through the given pixel.

[0177] In step 1808, the object recognition module 214 uses an artificial neural network to recognize objects in the two-dimensional panoramic image. In some embodiments, the artificial neural network is used to recognize objects in the two-dimensional image. A two-dimensional image may include one or more recognized objects. The same recognized object may be recognized in different two-dimensional images from different viewpoints. In some embodiments, the object recognition module 214 may generate a boundary polygon that depicts the boundary or outer limit of the recognized object. An example of a boundary polygon is a Fig. 17B 1722. In some embodiments, the object recognition module 214 recognizes a 2D image subset of the received 2D images for each recognized object. Each image of the 2D image subset includes pixels depicting a particular object found in the physical environment. A particular object may be found in more than one 2D image. Each 2D image may include more than one object. Each 2D object detection represents an instance of a physical object viewed by the image capture device. Examples of 2D object detections are Fig.17A 3D object detection 1712 depicts a bounding box of an instance of a recognized object, in this example a chair.

[0178] In step 1810, the object recognition module 214 may associate each pixel of the two-dimensional object detection with a surfel or surface of the three-dimensional model of the physical environment obtained from the estimated depth data from step 1806. In some embodiments, the object recognition module 214 associates clusters of pixels of the two-dimensional object detection with surfels or surfaces of the three-dimensional model of the physical environment obtained from the estimated depth data from step 1806. The estimated depth data may be utilized to determine the location of a particular two-dimensional object in the three-dimensional model. The object recognition module 214 may recognize a unique object in the three-dimensional model of the physical environment based on the location of each two-dimensional object detection.

[0179] In step 1814, the target recognition module 214 can provide a cumulative confidence of a specific target identified in the three-dimensional target detection set based on the three-dimensional target probability. The cumulative confidence can be aggregated, averaged, weighted averaged, etc. The target recognition module 214 can send a request to the target data storage 220 to update or create a target instance entry for a specific target. The target recognition module 214 can send a request to the target data storage 220 to update or create a target type entry associated with a specific target type. In addition, the target recognition module 214 can send a request to the tag module 216 to define a tag and a specific point in the three-dimensional model to associate the tag. The target instance entry can contain or indicate a specific instance of a target type. The target type entry can be associated with a specific target type. In some embodiments, the target data storage 220 can store a target type number indicating the total number of targets of the target type on the floor of the 3D model, a portion of the 3D model, and / or the entire 3D model. The target recognition module 214 can increase the incremental count of the target type entry associated with the restaurant chair.

[0180] Fig.21 A flow diagram of an object detection process 2100 is depicted in accordance with some embodiments. In step 2102, the communication module 204 receives a structured image set from the image capture device 106A. Generating the structured image set may be a result of a capture process of a Matterport's Pro3 camera. The structured image set generated by Matterport's capture process generates a depth map corresponding to each image in the image set. In some embodiments, step 2102 includes color processing to reduce image artifacts. The structured image set may include panoramic two-dimensional images, three-dimensional panoramic images, and depth maps of the physical environment.

[0181] In step 2104, the object recognition module 214 includes a Mask R-CNN for detecting and recognizing two-dimensional objects in a digital image. The object recognition module 214 can further recognize image pixels associated with the two-dimensional object. These recognized pixels can be referred to as object pixels. In some embodiments, the Mask R-CNN can provide a two-dimensional object probability, that is, a probability that a particular object in each of the subsets of two-dimensional object detections belongs to a particular category of two-dimensional objects.

[0182] An example of Mask R-CNN can be found in https: / / arxiv.org / abs / 1703.06870 Found in . MaskR-CNN takes a color image as input and generates a set of pixel "masks" and categorical "logical values" for each object it detects in the image, which is an estimate of the likelihood that the object belongs to each of a fixed set of object semantic categories that the network has been trained to detect. In some embodiments, the object recognition module 214 can generate a boundary polygon that delineates the boundaries or outer limits of the recognized objects.

[0183] In step 2106, for each two-dimensional object detected in step 2104, the object recognition module 214 may associate the object pixels with a facet or surface of the three-dimensional model of the physical environment to obtain a three-dimensional object detection. The object recognition module 214 may provide a three-dimensional object probability, i.e., a three-dimensional object probability that a particular object in each of the subset of three-dimensional object detections belongs to a particular class.

[0184] In step 2108, object recognition module 214 may merge three-dimensional object detections corresponding to the same object. In some embodiments, object recognition module 214 may identify three-dimensional object detections to be merged based on spatial properties of each three-dimensional object detection, such as the location of each three-dimensional object in the three-dimensional model of the physical environment.

[0185] In addition to combining 3D object detections, object recognition module 214 may also provide a cumulative confidence that a particular object belongs to a particular object class based on the 3D object probability of each 3D object detection.

[0186] In step 2110 , the object recognition module 214 provides the cumulative confidence to the user interface. Fig. 20 An example is shown. A user may interact with an example user interface 2000 that depicts an area of ​​a physical environment and an icon depicting a location space in which an object is located.

[0187] Fig. 22 is a block diagram illustrating an example machine capable of reading instructions from a machine-readable medium and executing those instructions in a processor to perform the machine processing tasks discussed herein, such as the engine operations discussed above. Specifically, Fig. 22 A diagram of a machine in the example form of a computer system 2200 is shown within which instructions 2224 (e.g., software) for causing the machine to perform any one or more of the methodologies discussed herein may be executed. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines, such as through the Internet. In a networked deployment, the machine may operate in the capacity of a server or a client user machine in server-client user network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0188] The machine may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a smart phone, a network appliance, a network router, a switch or a bridge, or any machine capable of executing instructions 2224 (sequentially or otherwise) that specify actions to be taken by the machine. In addition, while only a single machine is shown, the term "machine" should also be construed to include any collection of machines that individually or jointly execute instructions 2224 to perform any one or more of the methodologies discussed herein.

[0189] The example computer system 2200 includes a processor 2202 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio frequency integrated circuits (RFICs), or any combination thereof), a main memory 2204, and a static memory 2206, which are configured to communicate with each other via a bus 2208. The computer system 2200 may also include a graphics display unit 2210 (e.g., a plasma display panel (PDP), a liquid crystal display (LCD), a projector, or a cathode ray tube (CRT)). The computer system 2200 may also include an alphanumeric input device 2212 (e.g., a keyboard), a cursor control device 2214 (e.g., a mouse, a trackball, a joystick, a motion sensor, or other pointing instrument), a data storage 2216, a signal generation device 2218 (e.g., a speaker), an audio input device 2226 (e.g., a microphone), and a network interface device 2220, which are also configured to communicate via the bus 2208.

[0190] The data storage 2216 includes a machine-readable medium 2222 on which are stored instructions 2224 (e.g., software) embodying any one or more of the methods or functions described herein. The instructions 2224 (e.g., software) may also reside completely or at least partially within the main memory 2204 or within the processor 2202 (e.g., within the processor's buffer memory) during execution of the instructions 2224 (e.g., software) by the computer system 2200, the main memory 2202 and the processor 2202 also constituting machine-readable media. The instructions 2224 (e.g., software) may be sent or received over a network (not shown) via the network interface 2220.

[0191] Although the machine-readable medium 2222 is shown as a single medium in the example embodiment, the term "machine-readable medium" should be taken to include a single medium or multiple media (such as a centralized or distributed database, or associated caches and servers) that can store instructions (such as instructions 2224). The term "machine-readable medium" should also be taken to include any medium that can store instructions (such as instructions 2224) executed by a machine and cause the machine to perform any one or more of the methods disclosed herein. The term "machine-readable medium" includes, but is not limited to, data repositories in the form of solid-state memory, optical media, and magnetic media.

[0192] In this specification, the term "module" refers to computing logic used to provide specified functionality. Modules can be implemented in hardware, firmware, and / or software. In the case where the modules described herein are implemented as software, the module can be implemented as a stand-alone program, but can also be implemented in other ways, such as as part of a larger program, as any number of separate programs, or as one or more static or dynamic link libraries. It should be understood that the named modules described herein represent one embodiment and other embodiments may include other modules. In addition, other embodiments may lack the modules described herein and / or distribute the described functions among the modules in different ways. In addition, functions attributed to more than one module may be merged into one module. In embodiments of modules implemented by software, they are stored on a computer-readable persistent storage device (such as a hard disk), loaded into a memory, and executed by one or more processors, as described above in conjunction with Fig. 22 Alternatively, the hardware or software modules may be stored elsewhere within the computing system.

[0193] As described herein, a computer or computing system includes hardware elements for the operations described herein, regardless of Fig. 22 The specific references to these elements in the text include, for example, one or more processors, high-speed memory, hard disk storage and backup, network interfaces and protocols, input devices for data input, and output devices for display, printing or other data presentation. There may be many variations of the system architecture described herein. The entities of these systems and their respective functions may be combined or redistributed.

Claims

1. A system, wherein include: one or more processors; as well as a memory containing instructions to control one or more of the processors to: receiving a plurality of images representing an interior of a physical environment; using a neural network for object recognition to recognize a first object in a first image of the plurality of images, the first object being associated with first object location information indicating a location relative to the physical environment; recognizing a second object in a second image of the plurality of images using the neural network for object recognition; Determining whether the first target in the first image and the second target in the second image are located near or at similar locations in the physical environment based on location information associated with the first image and second target location information associated with the second image; If the first object and the second object are located nearby or similarly positioned, then the first object and the second object are instances of a single object; storing the similar locations associated with the single target; displaying an interactive tour visualization of a 3D model of the physical environment, the 3D model including the single object; receiving a target information request regarding a target location through the interactive tour visualization; as well as The similar locations of the single target are provided for display in the interactive tour visualization.

2. The system according to claim 1, in, The instructions further control the one or more processors to determine whether the first target and the second target are the same target type.

3. The system according to claim 2, in, If the first object and the second object are located nearby or similarly positioned, and the first object and the second object are of the same object type, then the first object and the second object are instances of the single object.

4. The system according to claim 2, in, Recognizing the first object using the neural network for object recognition also includes classifying the first object using the neural network to recognize an object type of the first object.

5. The system according to claim 1, in, The instructions further control the one or more processors to determine the first target positioning information and the second target positioning information based on a mesh of the 3D model.

6. The system according to claim 1, in, The first image includes a position and an orientation of an image capture device that captured the first image in the physical environment, and the instructions further control the one or more processors to determine the first target location information based on the position and orientation of the image capture device that captured the first image.

7. The system according to claim 6, in, The second image includes a position and an orientation of an image capture device that captured the second image in the physical environment, and the instructions further control the one or more processors to determine the second object positioning information based on the position and orientation of the image capture device that captured the second image.

8. The system according to claim 1, in, The first image is associated with depth data of the physical environment, the depth data being generated by a depth data device, and the first object location is determined based on the first object depicted in the first image and the associated relevant depth data.

9. The system of claim 8, wherein the depth data device is a lidar.

10. A non-transitory computer readable medium comprising executable instructions executable by one or more processors to implement a method, the method include: receiving a plurality of images representing an interior of a physical environment; recognizing a first object in a first image of the plurality of images using a neural network for object recognition, the first object being associated with first object location information indicating a location relative to the physical environment; recognizing a second object in a second image of the plurality of images using the neural network for object recognition; determining whether the first target in the first image and the second target in the second image are located near or at similar locations in the physical environment based on location information associated with the first image and second target location information associated with the second image; If the first object and the second object are located nearby or similarly positioned, then the first object and the second object are instances of a single object; storing the similar locations associated with the single target; displaying an interactive tour visualization of a 3D model of the physical environment, the 3D model including the single object; receiving a target information request regarding a target location through the interactive tour visualization; as well as The similar locations of the single target are provided for display in the interactive tour visualization.

11. The non-transitory computer readable medium of claim 10, the executable instructions being executable by the one or more processors to further: A determination is made as to whether the first target and the second target are of the same target type.

12. The non-transitory computer readable medium of claim 11, in, If the first object and the second object are located nearby or similarly positioned, and the first object and the second object are of the same object type, then the first object and the second object are instances of the single object.

13. The non-transitory computer readable medium of claim 11, in, Recognizing the first object using the neural network for object recognition also includes classifying the first object using the neural network to recognize an object type of the first object.

14. The non-transitory computer readable medium of claim 10, the executable instructions being executable by the one or more processors to further: The one or more processors are controlled to determine the first object positioning information and the second object positioning information based on a mesh of the 3D model.

15. The non-transitory computer-readable medium of claim 10, wherein the first image includes a position and an orientation of an image capture device that captured the first image in the physical environment, and the executable instructions are executable by the one or more processors to further control the one or more processors to determine the first target positioning information based on the position and orientation of the image capture device that captured the first image.

16. The non-transitory computer readable medium of claim 15, in, The second image includes a position and an orientation of an image capture device that captured the second image in the physical environment, and the executable instructions are executable by the one or more processors to determine the second object positioning information based on the position and orientation of the image capture device that captured the second image.

17. The non-transitory computer readable medium of claim 10, in, The first image is associated with depth data of the physical environment, the depth data being generated by a depth data device, and the first object location is determined based on the first object depicted in the first image and the associated relevant depth data.

18. The non-transitory computer readable medium of claim 17, wherein the depth data device is a lidar.

19. A method, include: receiving a plurality of images representing an interior of a physical environment; recognizing a first object in a first image of the plurality of images using a neural network for object recognition, the first object being associated with first object location information indicating a location relative to the physical environment; recognizing a second object in a second image of the plurality of images using the neural network for object recognition; determining whether the first object in the first image and the second object in the second image are located near or at similar locations in the physical environment based on location information associated with the first image and location information of a second object associated with the second image; If the first object and the second object are located nearby or similarly positioned, then the first object and the second object are instances of a single object; storing the similar locations associated with the single target; displaying an interactive tour visualization of a 3D model of the physical environment, the 3D model including the single object; receiving a target information request regarding a target location through the interactive tour visualization; as well as The similar locations of the single target are provided for display in the interactive tour visualization.

Citation Information

Patent Citations

  • Visual organization of information via associated geospatial data

    CN102792322A

  • Building virtual reality (VR) gaming environments using real-world virtual reality maps

    CN110227266A

  • Utilizing deep neural network-based model to identify visually similar digital images based on user-selected visual attributes

    CN110503124A

  • System and method for generating combined embedded multi-view interactive digital media representations

    US20210344891A1

  • Methods and systems for providing an immersive virtual reality experience

    US20220004254A1