Automated determination of image acquisition locations within a building interior using multiple data capture devices

Abstract

Techniques are described for automatically determining a capture location of an image from a camera device using multiple devices, such as within a building interior and using data from multiple devices (e.g., visual data from the image of the camera device and additional data captured by a separate mobile computing device in the same area as the camera device), and then using the determined image capture location information in one or more further automated ways. The image may be a panoramic image or another type, and the determined capture location of such an image may be at least a location on a floor plan of a building, and additionally, the automated image capture location determination may further be performed without having or using information from any depth sensor or other distance measuring device regarding the distance from the capture location of the image to walls or other objects in the surrounding building.

Description

Technical Field

[0001] The following disclosure generally relates to technology for automatically determining the capture location of an image within a building's interior by using data captured from multiple devices and then using the determined capture location information in one or more ways to determine an image of the interior of a building's room on a floor plan of the building based on a combination of visual data captured from a camera device and additional data captured from a separate mobile computing device and using the determined image location to improve navigation of the building. Background Art

[0002] In various fields and situations (such as, building analysis, property inventory, real estate acquisition and development, remodeling and renovation services, general contracting, and other situations), it may be desirable to view information about the interior of a house, office, or other building without having to physically visit or enter the building, including determining actual as-built information about the building rather than design information obtained before the building was built. However, it may be difficult to effectively capture, represent, and use such building interior information, including displaying visual information captured within the building interior to a user at a remote location (e.g., enabling the user to fully understand the layout and other details of the interior, including controlling the display in a manner selected by the user). In addition, although building floor plans can provide some information about the layout and other details of the building interior, in some cases, such use of floor plans has some disadvantages, including that floor plans may be difficult to build and maintain, difficult to accurately scale and populate information about the interior of a room, difficult to visualize and otherwise use, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1A to Figure 1B is a diagram depicting an exemplary building interior environment and computing system for use in embodiments of the present disclosure, such as for performing automated operations to determine a capture location of an image of a building interior by using data captured from multiple devices and subsequently using the determined capture location information in one or more ways.

[0004] Figures 2A to 2U Examples of automated operations for generating and presenting information about a floor plan of a building based on images taken within the building's interior are described, thereby automatically determining and presenting acquisition locations of such images using data captured from multiple devices.

[0005] Figure 3 is a block diagram illustrating a computing system suitable for executing embodiments of one or more systems that perform at least some of the techniques described in this disclosure.

[0006] Figure 4An example flow chart of an image capture and analysis (ICA) system routine is illustrated in accordance with an embodiment of the present disclosure.

[0007] FIG. 5A to FIG. 5B An example flow chart for a Mapping Information Generation Manager (MIGM) system routine according to an embodiment of the present disclosure is illustrated.

[0008] FIG. 6A to FIG. 6B An example flow diagram of an Image Location Determination Manager (ILDM) system routine is illustrated in accordance with an embodiment of the present disclosure.

[0009] Figure 7 An example flow diagram of a building map viewer system routine is illustrated in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0010] The present disclosure describes methods for using a computing device to perform automated operations related to determining an acquisition location of an image, such as within a building interior by using data captured from multiple devices, and subsequently using the determined image acquisition location information in one or more further automated manners. The image may, for example, include a panoramic image or other image (e.g., a rectilinear stereo image) acquired at an acquisition location in or around a multi-room building (e.g., a house, office, etc.), generally referred to herein as a "target image", and the determined image acquisition location information of such target image may include at least a location on a floor plan of the building and in some cases also include an orientation or other direction information (e.g., a global compass direction) of at least a portion of the target image, and in addition, in at least some such embodiments, the automated image acquisition location determination is further performed without having or using information from any depth sensor or other distance measurement device regarding the distance from the acquisition location of the target image to walls or other objects in the surrounding building. In various embodiments, the determined image acquisition location information for one or more target images acquired for a building may be further used in various ways, such as in conjunction with a corresponding building floor plan and / or other generated mapping-related information for the building, including for controlling a mobile device (e.g., an autonomous vehicle), for display or other presentation in a corresponding GUI (graphical user interface) on one or more client devices via one or more computer networks, etc. Additional details regarding automated determination and use of image acquisition location information are included below, and in at least some embodiments, some or all of the techniques described herein may be performed via automated operation of an Image Location Determination Manager ("ILDM") system, as discussed below.

[0011] As described above, automated operation of the ILDM system may include determining the acquisition position and optionally orientation of a target panoramic image (or other target image) captured by a camera device in a room of a house or other building (or in another defined area) by automatically using the visual data of the target image and additional data captured from one or more other nearby devices, the combination of the acquisition position and orientation of the target image sometimes being referred to herein as the "pose" or "acquisition position" or simply "position" of the target image. Such data captured from multiple devices and used for automated determination of the acquisition position of the target image may include visual data in the target image acquired by the camera device, and additional data acquired by a separate mobile computing device near the camera device (e.g., a mobile vehicle / device carried by the same user or carrying the camera device), such as additional visual data in one or more other images acquired by the mobile computing device at one or more locations in the same room where the target image was acquired and / or additional acquisition metadata acquired by the mobile computing device related to the acquisition of other images (e.g., data from one or more IMUs or inertial measurement units of the mobile computing device). In at least some embodiments, additional data captured by the mobile computing device is used to determine a position (or pose) of the mobile computing device when acquiring one or more other images, and the acquisition position of the camera device for the target image is determined at least in part relative to the determined position of the mobile computing device, as discussed in more detail below.

[0012] In at least some embodiments, the determined position of the mobile computing device is based at least in part on performing SLAM (simultaneous localization and mapping) and / or SfM (structure from motion) and / or MVS (multi-view stereo) analysis, such as using motion data from an IMU sensor on the mobile computing device in combination with visual data from one or more image sensors on the mobile computing device, including in at least some such embodiments using additional data captured by the mobile computing device to generate an estimated three-dimensional (“3D”) shape of the enclosed room (e.g., based on a 3D point cloud having multiple 3D data points and / or estimated planar surfaces having walls and optionally a floor and / or ceiling), in some such embodiments, these automated operations are performed without using any depth sensors or other distance measuring devices from the mobile computing device to walls or other objects in the surrounding room, while in other embodiments, the mobile computing device (or other additional associated mobile devices) may capture depth data to the walls of the surrounding room and use the captured depth data as part of determining the position of the mobile computing device. In some embodiments, automated determination of the position of the mobile computing device may be further performed as part of generating a path of travel of the mobile computing device through the enclosed room (e.g., using one or more of SLAM, SfM, and / or MVS analysis), whether instead of or in addition to generating a 3D shape of the enclosed room, in other embodiments, automated determination of the position of the mobile computing device may be based at least in part on other analysis, such as via Wi-Fi triangulation, visual inertial odometry (“VIO”), etc. The following includes additional details about automated operations that, in at least some embodiments, may be performed by the ILDM system to determine a captured position and, optionally, an orientation of a mobile computing device in a room using one or more types of captured data.

[0013] In addition to automated operations for analyzing additional data captured by a separate mobile computing device to determine a first estimated room shape (e.g., a 3D room shape) of a room enclosing a capture location of a target panoramic image (or other target image) captured by the camera device, automated operations of the ILDM system may also include determining an additional estimated room shape (e.g., an additional 3D room shape) of the enclosed room based at least in part on an analysis of visual data in the target image, such as based at least in part on performing MVS (multi-view stereo) and / or visual odometry (“VO”) analysis, such as without using any depth sensors or other distance measuring devices from the camera device to walls or other objects in the surrounding room, in some embodiments, using the visual data of the target image to determine the additional estimated room shape of the enclosed room may also include using data from one or more IMU sensors of the camera device (e.g., using SLAM and / or SfM techniques), but in other embodiments, using the visual data of the target image to determine the additional estimated room shape of the enclosed room may not include using any such other IMU data (e.g., no data other than the visual data of the target image may be used). For example, in at least some embodiments, the described techniques include using one or more trained neural networks or other techniques to estimate a 3D room shape shown in a target image, as non-exclusive examples, such 3D room shape estimation may include one or more of the following: using a trained convolutional neural network or other analysis technique to take a target image as input and estimate a 3D point cloud of walls and other surfaces of an enclosed room from the visual content of the target image and / or estimate a piecewise planar representation of the enclosed room (e.g., 3D walls and other planar surfaces) from the visual content of the target image; using a trained neural network or other analysis technique to take a target image as input and estimate a wireframe structure of an enclosed room (e.g., showing walls) from the visual content of the target image; The method comprises detecting one or more structural lines of the target image (for example, a boundary between walls, a boundary between walls and ceilings, a boundary between walls and floors, outlines of doorways and / or other wall openings between rooms, outlines of windows, etc.); using a trained neural network or other analysis technique to detect wall structural elements in the visual content of the target image (for example, windows and / or skylights; passages in and out of a room, such as doorways and other openings in walls, stairs, corridors, etc.; boundaries between adjacent walls; boundaries between walls and floors; boundaries between walls and ceilings; corners (or solid geometric vertices) where at least three surfaces or planes intersect; etc.) and optionally detecting other fixed structural elements (for example, countertops, bathtubs, sinks, islands, fireplaces, etc.) and optionally generating 3D bounding boxes for the detected elements; etc.Included below are additional details regarding automated operations that, in at least some embodiments, may be performed by an ILDM system to identify an estimated room shape of a room enclosing a target image based at least in part on visual data from the target image.

[0014] Given a target image captured by a camera device in an enclosed room and additional data captured by a mobile computing device in the enclosed room, automated determination of a capture location and optionally a capture orientation of the target image may include performing one or more automated operations including using the automatically determined location and optionally the orientation of the mobile computing device in the enclosed room at one or more times, and including optionally using one or more determined estimated room shapes of the enclosed room from visual data from the target image and the additional data from the mobile computing device. Non-exclusive examples of such automated operations for determining a capture location and optionally a capture orientation of the camera device in the enclosed room for the target image based at least in part on the additional data captured by the mobile computing device in the enclosed room include the following:

[0015] (a) analyzing additional visual data of one or more other images captured by the mobile computing device to identify a camera device in the additional visual data, and using the position of the identified camera device in the additional visual data as part of determining a capture position, and optionally a capture orientation, of the camera device in the enclosed room; and / or

[0016] (b) analyzing additional visual data of one or more other images captured by the mobile computing device to identify one or more markers located on the camera device in the additional visual data, and using one or more locations of the identified one or more markers in the additional visual data as part of determining a capture position and, optionally, a capture orientation of the camera device in the enclosed room; and / or

[0017] (c) analyzing visual data of the target panoramic image (or other target images) to generate a first estimated shape of the enclosed room, analyzing additional visual data of one or more other images captured by the mobile computing device to generate a second estimated shape of the enclosed room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape at which the visual data of the target panoramic image was captured using offsets of corresponding portions of the shapes; and / or

[0018] (d) analyzing visual data of the target panoramic image (or other target image) to generate a first estimated shape of the enclosed room, analyzing depth data captured by the mobile computing device in the enclosed room to generate a second estimated shape of the enclosed room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape at which the visual data of the target panoramic image was captured using offsets of corresponding portions of the shapes; and / or

[0019] (e) analyzing visual data of the target panoramic image (or other target image) and additional visual data of one or more other images captured by the mobile computing device to identify features (e.g., 2D features) visible in both the visual data and the additional visual data, and using offsets in the positions of the identified features as part of determining a capture position and, optionally, a capture orientation of the camera device in the enclosed room; and / or

[0020] (f) analyzing movement data of the mobile computing device in the enclosed room to identify the location at which the camera device captures the target panoramic image as the mobile computing device moves to and / or from the location, and using the identified location as part of determining a capture position and, optionally, a capture orientation of the camera device in the enclosed room.

[0021] Additionally, other non-exclusive examples of such automated operations for using other types of information to determine a capture position and optionally a capture orientation of a camera device for a target image in an enclosed room include the following:

[0022] (g) analyzing visual data of the target panoramic image and other visual data of another panoramic image captured in the enclosed room (e.g., another previously acquired target image) to identify features (e.g., 2D features) visible in both the target panoramic image and the other panoramic image, and using offsets of the positions of the identified features to determine a common coordinate system for the target panoramic image and the other panoramic image for combining the visual data and the other visual data and determining an acquisition position and, optionally, an acquisition orientation of a camera device for the target image in the enclosed room (e.g., in a manner relative to a previously determined acquisition position and, optionally, an acquisition orientation of the other panoramic image); and / or

[0023] (h) analyzing visual data of a target panoramic image (or other target image) to generate a first estimated shape of an enclosed room, retrieving a second previously determined shape of the enclosed room (e.g., retrieved from a partially or fully defined floor plan of a building in which the enclosed room is located), and comparing the first room shape to the second room shape to identify a location within the second room shape at which the visual data of the target panoramic image was captured using offsets of corresponding portions of the shapes.

[0024] In addition, the automatically determined acquisition position and, optionally, acquisition orientation of the camera device for the first target image in the first enclosed room can be combined with corresponding information determined for one or more second target images in one or more second enclosed rooms (whether the same or different rooms as the first enclosed room), such as using the above-described type (f) automated operations, so as to create a single common coordinate system to connect some or all of the target images acquired by the camera device in the building, including linking the acquisition positions of the target images, as described in more detail elsewhere in this document. In addition, in at least some embodiments, for some or all of the above-described types of automated operations for determining the acquisition position and, optionally, acquisition orientation of the camera device for the target image in the enclosed room, the results of that type of automated operation may also include information about the accuracy and / or likelihood and / or uncertainty of those results, such as generating and providing one or more confidence levels for the results of each of those types of automated operations.

[0025] If, in a given embodiment and scenario, only a single type of automated operation is performed to determine the acquisition position and, optionally, the acquisition orientation of a camera device for a target image in a closed room, the result of that single automated operation may be selected and used to represent the acquisition position and, optionally, the acquisition orientation of the target image. Alternatively, if, in a given embodiment and scenario, multiple types of automated operations are performed to determine the acquisition position and, optionally, the acquisition orientation of a camera device for a target image in a closed room, the results of those multiple automated operations may be used in various ways to represent the acquisition position and, optionally, the acquisition orientation of the target image. For example, in some such embodiments and scenarios, a single type of result from the multiple types of automated operations may be selected and used to represent the acquisition position and, optionally, the acquisition orientation of the target image, such as for the result with the highest confidence level and / or based on a defined priority of some or all of the multiple types of automated operations (e.g., using the result of the highest priority type of automated operation, such as operations (b) and / or (d), if performed and available, otherwise using the result of the next highest priority type of automated operation performed and available). Alternatively, in some embodiments and scenarios, the results of multiple types of automated operations can be combined and used together to represent the acquisition position and optionally the acquisition orientation of the target image, such as performing an average (e.g., a weighted average weighted using a confidence level associated with a particular result), performing a statistical analysis (e.g., including discarding outliers at the lower and / or upper ends of the confidence values), etc. In addition, in some embodiments and scenarios, the results of one or more types of automated operations (e.g., operations (a) and / or (b) and / or (f)) can be used as initial values, which are provided as input to one or more additional types of automated operations (e.g., operations (c) and / or (d) and / or (e)) that update (e.g., refine) those initial values, where the updated values ​​of one or more of the additional types of automated operations are used to represent the acquisition position and optionally the acquisition orientation of the target image. The following includes additional details about automated operations, in at least some embodiments of one or more types of automated operations, which can be performed by the ILDM system to determine the acquisition position and optionally the acquisition orientation of the camera device for the target image in the enclosed room.

[0026] In various embodiments, the described techniques provide various benefits, including allowing partial or complete floor plans of multi-room buildings and other structures to be automatically generated simultaneously with the acquisition of one or more target images acquired for the building or other structure, and / or allowing such existing partial or complete floor plans to be augmented with information about the acquisition locations in the building or other structure at which the target images were acquired, including in embodiments without having to have or using information from a depth sensor or other distance measuring device about the distance from the image acquisition location to walls or other objects in the surrounding building or other structure.Non-exclusive examples of such benefits include the following: the ability to provide feedback during capture of one or more target images captured for a building or other structure (e.g., displaying or otherwise providing to a user a determined room shape of an enclosed room indicating the capture location and optionally the capture orientation of each of the one or more target images, such as a portion of a partial or complete floor plan of a building or other structure), including optionally allowing a user to determine and indicate one or more other areas of a building where one or more other target images are to be captured (e.g., for a partial floor plan, capturing additional target images in other areas of the building not already represented in the partial floor plan); the ability to interconnect multiple target images and displaying at least one of the target images in a direction of the other linked target image with a user-selectable visual indicator that, when selected, causes display of a corresponding other of the linked target images, such as placing the various target images in a common coordinate system showing their relative positions, or otherwise determining at least a direction between a pair of target images (e.g., based at least in part on automated analysis of visual content of the target images in the pair, such as in a manner similar to operation (f) above, and optionally based on other movement data from the mobile computing device along the path of travel between the target images), and linking the various target images using the determined inter-image direction; capable of being displayed on the mobile computing device The invention also provides a method for determining a capture position and optionally a capture orientation of a target image and / or a 3D room shape of an enclosed room for the target image, such as in a near real-time or quasi-real-time manner relative to the capture of the target image, and optionally immediately using the initial information determined on the mobile computing device (e.g., for display to a user on the mobile computing device), while also further supplying to one or more other computing devices (e.g., a remote server computing system) for updating (e.g., refinement); capable of analyzing visual data of the target image to detect objects of interest (e.g., structural wall elements such as windows, doorways and other wall openings, etc.) in the enclosed room and determining whether those detected objects are at least partially or completely within the enclosed room; The invention can also be used to determine a position in a determined room shape based entirely on additional data captured by a mobile computing device; can analyze additional data captured by the mobile computing device (e.g., movement data from one or more IMU sensors, visual data from one or more image sensors, etc.) to determine a travel path of the mobile computing device in multiple rooms, to identify wall openings (e.g., doorways, stairwells, etc.) of multiple rooms based at least in part on the additional data (and optionally based on visual data of one or more target images captured in one or more rooms), and optionally further use such information about the identified wall openings to position the determined 3D room shapes of the multiple rooms together; etc.

[0027] Furthermore, the described automated techniques allow such image acquisition location information to be determined more quickly than previously existing techniques, and in at least some embodiments with greater accuracy, including by using information acquired from the actual building environment (rather than from plans of how the building should theoretically be built), and enabling the capture of structural element changes that occur after the building is initially constructed. Such described techniques further provide the benefit of allowing improved automated navigation of a building by a mobile device (e.g., a semi-autonomous or fully autonomous vehicle) based at least in part on the determined image acquisition locations, including significantly reducing the computing power and time used to attempt to otherwise learn the layout of the building. Additionally, in some embodiments, the described techniques can be used to provide an improved GUI in which a user can more accurately and more quickly obtain information about the interior of a building (e.g., for use in navigating the interior), including in response to a search request, as part of providing personalized information to a user, as part of providing a user with value estimates and / or other information about a building, etc. The described techniques also provide a variety of other benefits, some of which are further described elsewhere herein.

[0028] As described above, in some embodiments, a building floor plan with associated room shape information for some or all of the rooms of the building can be used in at least some embodiments for automated determination of a target image acquisition location within the building, in at least some such cases, the 2D and / or 3D room shapes of the rooms (or other defined areas) shown on the floor plan may have been previously automatically determined, while in other cases, some or all of the room shapes of the rooms (or other shapes of the defined areas) may be determined in other ways (e.g., automatically determined simultaneously when determining the acquisition location of the target image, previously or simultaneously determined based at least in part on manual input by one or more users, etc.). In various embodiments, the building floor plan with associated room shape information can have various forms, such as a 2D (two-dimensional) floor map of the building (e.g., an orthogonal top view or other overhead view of a schematic floor map that does not include or display height information) and / or a 3D (three-dimensional) or 2.5D (2.5-dimensional) floor map model of the building that displays height information. In at least some embodiments, automated determination of a capture location of a target image within a multi-room building may generally include retrieving determined room shapes for one, some, or all of the rooms of the building, and optionally identifying one of those rooms whose determined room shape best matches the room shape of a target image, the room shape of the target image being estimated from visual content of the target image and being placed (e.g., sized and rotated) to match the determined room shape of the identified room, including comparing aspects of the visual content of the target image with the determined room shape of the identified room to at least determine the location within the identified room at which the target image was captured. Once the estimated room shape of the target image of the enclosed room is automatically identified, it may be compared to candidate room shapes (e.g., previously determined room shapes for one, some, or all of the rooms of the building) to automatically determine a capture location and optionally a capture orientation of the target image within the determined candidate room shapes (e.g., the candidate that best matches the estimated room shape of the target image), wherein the room having such a determined room shape is referred to herein as the "target" room within the building in which the capture location of the target image occurs.

[0029] In addition, in some embodiments of images captured in defined areas other than rooms, the estimated shape of the image of the defined area can be matched to the shapes of other defined areas in various ways. For example, in at least some embodiments, the target image (and optionally additional images) can be collected on the exterior of one or more buildings, such as in one of multiple separate areas of one or more properties (e.g., house, garden, patio, deck, backyard, side yard, front yard, pool, garage, dock, etc.) each having a previously or simultaneously determined area shape (e.g., 3D shape, 2D shape, etc.), if so, the shape of such other defined areas and the techniques described herein can be similarly used to automatically determine the capture location of the target image (and optionally additional images) with the shape of the area.

[0030] In some embodiments, the ILDM system's automated determination of the acquisition position of the target image captured in the room may also include additional operations, and the following includes corresponding additional details, including information about Figures 2A to 2U Examples and their associated descriptions.

[0031] As described above, in at least some embodiments and scenarios, some or all of the images captured for a building may be panoramic images each captured at one of a plurality of capture locations in or around the building, such as to generate a panoramic image at each such capture location from one or more of: a video captured at the capture location (e.g., a 360° video shot from a smartphone or other mobile device held by a user and rotated at the capture location), or multiple images captured from the capture location in multiple directions (e.g., from a smartphone or other mobile device held by a user and rotated at the capture location; from automated rotation of the device at the capture location, such as on a tripod at the capture location; etc.), or simultaneous capture of all image information for a particular capture location (e.g., using one or more fisheye lenses), etc. It will be appreciated that in some cases such panoramic images may be represented in a spherical coordinate system and provide up to 360° coverage around the horizontal and / or vertical axes (e.g., 360° coverage along the horizontal plane and around the vertical axis), while in other embodiments, the captured panoramic images or other images may include less than 360° vertical coverage (e.g., for images having a width to height exceeding typical aspect ratios, such as at or exceeding 21:9 or 16:9 or 3:2 or 7:5 or 4:3 or 5:4 or 1:1, including so-called "ultra-wide" lenses and resulting ultra-wide images). In addition, it will be appreciated that a user viewing such a panoramic image (or other images having sufficient horizontal and / or vertical coverage such that only a portion of the image is displayed at any given time) may be permitted to move the viewing direction to different orientations within the panoramic image so that a different subset of images (or "views") are rendered within the panoramic image, and in some cases, such a panoramic image may be represented in a spherical coordinate system (including if the panoramic image is represented in a spherical coordinate system and a particular view is rendered to convert the rendered image to a planar coordinate system, such as for a stereoscopic image view prior to display). In addition, acquisition metadata regarding the capture of such a panoramic image may be obtained and used in various ways, such as data acquired from an IMU sensor or other sensor of a mobile device as the mobile device is carried by a user or otherwise moved between acquisition locations, non-exclusive examples of such acquisition metadata may include one or more of the following: acquisition time; acquisition location, such as GPS coordinates or other location indication; acquisition direction and / or orientation; relative or absolute acquisition order of multiple images acquired or otherwise associated with a building; etc., and in at least some embodiments and scenarios, such acquisition metadata may also optionally be used as part of determining the acquisition location of the image, as further discussed below. The following includes additional details regarding automated operations involved in acquiring images and optionally acquiring metadata for an apparatus implementing an image capture and analysis (ICA) system, including Figure 1A to Figure 1B and FIG. 2A to FIG. 2Band additional details elsewhere in this article.

[0032] As also described above, in various embodiments, the shapes of the rooms of a building may be automatically determined in various ways, including, in some embodiments, at a time prior to automated determination of the acquisition location of a particular image within the building. For example, in at least some embodiments, a mapping information generation manager (MIGM) system may analyze various images acquired in and around a building to automatically determine the room shapes (e.g., 3D room shapes, 2D room shapes, etc.) of the rooms of the building and automatically generate a floor plan of the building. As an example, if multiple images are acquired within a particular room, those images may be analyzed to determine the 3D shape of the rooms of the building (e.g., to reflect the geometry of the surrounding structural elements of the building), the analysis may include, for example, automated operations to "register" the camera positions of the images in a common reference frame in order to "align" the images, and estimate the 3D positions and shapes of objects in the room, such as determining features visible in the content of such images (e.g., to determine the direction and / or orientation of the acquisition device when the particular image was taken, the path that the acquisition device traveled through the room, etc.), such as using SLAM techniques for multiple video frame images and / or using other SfM techniques for A set of "dense" images separated by at most a defined distance (such as 6 feet) to generate a 3D point cloud of the room, which includes 3D points along the walls of the room and at least some of the ceiling and floor of the room and optionally with 3D points corresponding to other objects in the room, etc.) and / or determine and aggregate information about the planes of detected features and the normal (orthogonal) directions of which planes to identify planar surfaces of possible locations of walls and other surfaces of the room and connect various possible wall locations (e.g., using one or more constraints, such as having 90° angles between walls and / or between walls and floors, as part of the so-called "Manhattan world assumption") and form an estimated room shape for the room. After determining the estimated room shapes of the rooms in the building, in at least some embodiments, the automated operations may also include locating multiple room shapes together to form a floor plan and / or other related mapping information of the building, such as connecting the various room shapes based, optionally at least in part, on information about doorways and stairwells and other inter-room wall openings identified in particular rooms and optionally, at least in part, on information about the determined path of travel of the mobile computing device between rooms. The following includes additional details regarding the automated operations involved in determining room shapes and combining room shapes to generate floor plans in an apparatus implementing the MIGM system, including details regarding Figure 1A to Figure 1B and Figure 2E to Figure 2N and additional details elsewhere in this article.

[0033] For illustrative purposes, some embodiments are described below in which particular types of information are collected, used, and / or presented in particular ways and using particular types of devices for particular types of structures—it will be understood, however, that the described techniques may be used in other ways in other embodiments, and that the invention is therefore not limited to the exemplary details provided. As a non-exclusive example, while a floor plan that does not include detailed measurements of particular rooms or the entire house may be generated for a house, it will be understood that in other embodiments, other types of floor plans or other mapping information may be similarly generated, including for buildings (or other structures or layouts) that are separate from the house. As another non-exclusive example, while a floor plan of a house or other building may be used for display to help a viewer navigate the building, in other embodiments, the generated mapping information may be used in other ways. As yet another non-exclusive example, while some embodiments discuss obtaining and using additional data from a mobile computing device that is separate from the camera device that captured the target image, in other embodiments, the one or more devices used in addition to the camera device may have other forms, such as using a mobile device that collects some or all of the additional data but does not provide its own computing capabilities (e.g., an additional "non-computing" mobile device), multiple separate mobile devices (whether mobile computing devices or non-computing mobile devices) that each capture some of the additional data, etc. In addition, the term "building" refers herein to any partially or completely enclosed structure, typically but not necessarily encompassing one or more rooms that visually or otherwise divide the interior space of the structure - non-limiting examples of such buildings include a house, an apartment building or individual apartments therein, a condominium, an office building, a commercial building or other wholesale and retail structures (e.g., a shopping mall, department store, warehouse, etc.), etc. The terms "collection" or "capture" as used herein with reference to a building interior, a collection location, or other location (unless the context clearly indicates otherwise) may refer to any recording, storage, or entry of media, sensor data, and / or other information related to spatial characteristics and / or visual characteristics and / or otherwise perceptible characteristics of a building interior, or a subset thereof, such as by a recording device or by another device that receives information from a recording device. As used herein, the term "panoramic image" may refer to a visual representation based on, including, or separable into, a plurality of discrete component images originating from substantially similar physical locations in different directions and depicting a larger field of view than any of the discrete component images depict alone, including images from a physical location having a viewing angle that is wide enough to include angles beyond what is perceptible by a person gazing in a single direction (e.g., greater than 120° or 150° or 180°, etc.).As used herein, the term "series" of collection locations generally refers to two or more collection locations, each of which is visited at least once in a corresponding order, regardless of whether other non-collection locations are visited between them, and regardless of whether the visits to the collection locations occur during a single continuous time period or at multiple different times, or by a single user and / or device or by multiple different users and / or devices. In addition, various details are provided in the drawings and text for illustrative purposes, but these details are not intended to limit the scope of the invention. For example, the sizes and relative positions of elements in the drawings are not necessarily drawn to scale, and some details are omitted and / or provided more prominently (e.g., via size and positioning) to enhance readability and / or clarity. In addition, the same reference numerals may be used in the drawings to identify similar elements or actions.

[0034] Figure 1A is an example block diagram of various devices and systems that may participate in the described techniques in some implementations. In particular, Figure 1A 1 illustrates a target panoramic image 165 that has been captured by one or more camera devices 186, such as relative to one or more buildings or other structures and under the control of an internal capture and analysis ("ICA") system 160, in which example the ICA system is executed on one or more server computing systems 180, wherein inter-image directional links may optionally have been generated for at least some of the image pairs (e.g., after performing automated acquisition positioning and optionally acquisition orientation determination for the target panoramic image using the ILDM system 140). Figure 1B An example of such a panoramic image acquisition location 210 for a particular house 198 is shown, as discussed further below, and additional details related to the automated operation of the ICA system are included elsewhere herein, including with respect to Figure 4 In at least some embodiments, at least some of the ICA system may be executed in part on a mobile computing device 185 that operates 187 in conjunction with one or more camera devices 186 (whether in addition to or in place of the ICA system 160 on one or more server computing systems 180), such as relative to target images captured by those camera devices and in the same room or other area as those camera devices during the capture of those target images to capture additional data, such as with respect to Figure 1B Further discussion. The MIGM (Mapping Information Generation Manager) system 160 is further described in Figure 1A to generate and provide building floor plans 155 and / or other mapping related information (e.g., room shape determination, linked panoramic images, etc.) based on the use of the panoramic images 165 and optionally associated metadata regarding their acquisition and linking, Figure 2M20 (referred to herein as "2-0" for clarity) shows an example of such a floor plan, as discussed further below, and elsewhere herein includes additional details related to the automated operation of the MIGM system, including with respect to FIG. 5A to FIG. 5B Additional details.

[0035] Figure 1A Further described is an ILDM (image location determination manager) system 140 that executes on one or more server computing systems 180 to determine a capture location and, optionally, a capture orientation of a target image 145 (e.g., a target panoramic image 165 obtained from an ICA system 160) captured in one or more building rooms, such as relative to a corresponding building floor plan 155 that was previously or simultaneously determined with the ILDM system's automated capture location and, optionally, capture orientation determination. In at least some embodiments and scenarios, one or more users of an ILDM client computing device 105 may further interact with the ILDM system 140 via a network 170, such as to assist in some automated operations of the ILDM system for determining a capture location of a target image based, at least in part, on an analysis of image content, and / or for subsequently using the determined capture location and, optionally, capture orientation information in one or more further automated manners. Additional details regarding the automated operations of the ILDM system are included elsewhere herein, including with respect to Figure 2P to Figure 2U and FIG. 6A to FIG. 6B In some embodiments, the ICA system and / or the MIGM system and / or the ILDM system 140 may be executed on the same server computing system, such as if multiple or all of those systems are operated by a single entity or otherwise executed in coordination with one another (e.g., some or all of the functionality in those systems are integrated together into a larger system), while in other embodiments, the ILDM system may instead obtain the target image and / or optionally other information (e.g., floor plan information, other additional images, etc.) from one or more external sources and optionally store them locally with the ILDM system (not shown) for further analysis and use.

[0036] One or more users (not shown) in one or more client computing devices 175 may further interact with the ILDM system 140 and optionally the ICA system and / or the MIGM system via one or more computer networks 170, such as obtaining and using the determined acquisition location and optionally acquisition orientation information, and / or obtaining and optionally interacting with the generated floor plan (on which one or more target images are already located), and / or obtaining additional information (such as one or more associated target images) and optionally interacting with it (e.g., changing between a floor plan view and a view of a particular target image at an acquisition location within or near the floor plan; changing the horizontal and / or vertical viewing direction according to which a corresponding view of a panoramic image is displayed, such as determining the portion of the panoramic image to which the current user viewing direction is directed, etc.). In addition, although Figure 1A Although not described in the , a floor plan (or a portion thereof) may be linked to or otherwise associated with one or more other types of information, including a floor plan of a multi-story or other multi-story building having multiple associated sub-floor plans of different floors or levels that are interconnected (e.g., via connected stairways), a two-dimensional ("2D") floor plan of a building linked to or otherwise associated with a three-dimensional ("3D") rendered floor plan of the building, etc. Figure 1A Not illustrated, but in some embodiments, the client computing device 175 (or other device, not shown) may receive and use the determined image capture location and optionally capture orientation information (optionally in combination with the generated floor plan and / or other generated mapping-related information) in additional manners, such as to control or assist in automated navigation activities of such devices (e.g., autonomous vehicles or other devices), either in lieu of or in addition to the display of the generated information.

[0037] exist Figure 1A In the depicted computing environment, network 170 may be one or more publicly accessible linked networks, such as the Internet, that may be operated by a variety of different parties. In other embodiments, network 170 may have other forms. For example, network 170 may instead be a private network, such as a corporate network or university network that is fully or partially inaccessible to non-privileged users. In still other embodiments, network 170 may include both private networks and public networks, wherein one or more of the private networks may access one or more of the public networks and / or be accessed from one or more of the public networks. Furthermore, in various scenarios, network 170 may include various types of wired and / or wireless networks. Additionally, client computing devices 175 and server computing systems 180 may include various hardware components and stored information, as described below with respect to Figure 3 Discuss in more detail.

[0038] exist Figure 1A In an example of , the ICA system 160 may perform automated operations involving generating a plurality of target panoramic images (e.g., each being a 360 degree panorama about a vertical axis) at a plurality of associated acquisition locations (e.g., in a plurality of rooms or other locations within a building or other structure and optionally around some or all of the exterior of the building or other structure), such as for generating and providing a representation of the interior of a building or other structure. The technique may further include: analyzing information to determine a relative position / orientation between each of two or more acquisition locations; creating an inter-panorama position / orientation link in the panorama to each of one or more other panoramas based on such determined position / orientation; and then, providing information to display or otherwise present a plurality of linked panoramic images of the respective acquisition locations within the building.

[0039] Figure 1B A block diagram of an exemplary building interior environment is depicted, in which target panoramic images have been generated and analyzed by a copy of an ILDM system (not shown) to determine acquisition locations and optionally acquisition orientations of those target images within a building 198 (in this example, a house 198), and optionally further analyzed by a copy of a MIGM system (not shown) to generate and provide (e.g., present) corresponding building floor plans and / or other mapping-related information (e.g., room shapes of rooms of a building, a set of linked target panoramic images, etc.). In particular, Figure 1BThe plurality of panoramic images are captured at a series of plurality of acquisition locations 210 associated with the premises 198 (e.g., the interior and exterior of the premises), such as by a user (not shown) carrying one or more camera devices 186 to capture target images and carrying one or more separate mobile computing devices 185 to capture additional data for the plurality of acquisition locations 210. An embodiment of the ICA system (e.g., the ICA system 160 on the server computing system 180; a copy of some or all of the ICA system executing on a user's mobile computing device, such as the ICA application system 155 executing in the memory 152 of the device 185; etc.) may automatically perform or assist in capturing data representing the interior of the building, and in some embodiments, further analyzing the captured data to generate linked panoramic images to provide a visual representation of the interior of the building. Although the user's mobile computing device may include various hardware components such as memory 152, display 142, one or more hardware processors 132, optionally one or more image sensors or other imaging systems 135, optionally one or more depth sensors or other distance measurement sensors 136, optionally one or more other sensors 148 (e.g., gyroscopes 148a, accelerometers 148b, compasses 148c, etc., such as part of one or more IMUs or inertial measurement units of the mobile device; altimeters; light detectors; etc.), optionally a GPS receiver, and optionally other components not shown (e.g., additional non-volatile storage devices; communication via network 170 and / or via direct device-to-device communication), the user's mobile computing device may include various hardware components such as memory 152, display 142, one or more hardware processors 132, optionally one or more image sensors or other imaging systems 135, optionally one or more depth sensors or other distance measurement sensors 136, optionally one or more other sensors 148 (e.g., gyroscopes 148a, accelerometers 148b, compasses 148c, etc., such as part of one or more IMUs or inertial measurement units of the mobile device; altimeters; light detectors; etc.), optionally a GPS receiver, and optionally other components not shown (e.g., additional non-volatile storage devices; communication via network 170 and / or via direct device-to-device communication). The mobile computing device 185 may include a transmission capability to communicate with other devices, such as an associated camera device 186, a remote server computing system 180, etc.; a microphone; one or more external lights; etc.), but in some embodiments, the mobile computing device does not include a distance measurement sensor 136 or is otherwise able to access or use other specialized equipment to measure the depth of objects in the building relative to the location of the mobile computing device, so that in such embodiments, the relationship between different target panoramic images and their capture locations can be determined based in part or in whole on matching elements in different images and / or by using information from other listed hardware components, but without using any data from any such distance measurement sensor 136. Although not illustrated for the sake of brevity, the one or more camera devices 186 may each similarly include at least one or more image sensors and storage for storing captured target images, and transmission capability to transmit the captured target images to other devices (e.g., an associated mobile computing device 185, a remote server computing system 180, etc.), optionally as well as one or more lenses and lights, and optionally in some embodiments, some or all of the other components of the mobile computing device are shown.Additionally, while direction indicators 109 are provided for viewer reference, in at least some embodiments, the mobile computing device and / or ICA system may not use such absolute direction information, such as instead determining relative directions and distances between panoramic images 210 without regard to actual geographic location or direction.

[0040] In operation, a user associated with mobile computing device 185 and camera device 186 arrives at a first acquisition location 210A within a first room within a building (in this example, via an entrance from exterior door 190-1 to a living room), and as camera device 186 is rotated about a vertical axis at the first acquisition location (e.g., where the user turns his or her body in a circle while keeping camera device 186 fixed relative to the user's body), a view of a portion of the building interior visible from the acquisition location 210A (e.g., some or all of the first room, and optionally small portions of one or more other adjacent or nearby rooms, such as through a doorway, hall, staircase, or other connecting passageway from the first room) is captured. Mobile computing device 185 may also optionally capture additional data at or near acquisition location 210A while rotating in the same manner as camera device 186 (e.g., additional visual data using imaging system 135, additional motion data using sensor module 148, additional depth data optionally using distance measurement sensor 136, etc.), although in some embodiments, such rotation of the mobile computing device may not be performed, and other such additional data may also be optionally captured as devices 185 and 186 move to and / or from the acquisition location. The actions of the mobile computing device 185 and the camera device 186 may be controlled or facilitated via the use of one or more programs executing on the mobile computing device 185 (e.g., via automated instructions to those devices or to another mobile device (not shown) that carries those devices through the building under its own power; via instructions to a user; etc.), such as the ICA application system 155 and / or optional browser 162, a control system 147 for managing I / O (input / output) and / or communications and / or networks for the device 185 (e.g., for receiving instructions from a user and presenting information to a user), etc., and the capture of visual data by the camera device and optionally the mobile computing device may be performed by recording video at location 210A and / or taking a series of one or more images at location 210A, including capturing visual information depicting a number of objects or other elements (e.g., structural details) visible in an image (e.g., a video frame) captured from or near the capture location. Figure 1BIn the example of , such objects or other elements include various elements that are structurally part of the walls of the rooms of the house (or structural "wall elements"), such as doorways 190 and 197 and their doors (e.g., with revolving doors and / or sliding doors), windows 196, wall boundaries (e.g., corners or edges) 195 (including corner 195-1 at the northwest corner of the house 198, corner 195-2 at the northeast corner of the first room, and corner 195-3 at the southwest corner of the first room), and in addition, in Figure 1B In the example of , such objects or other elements may further include other elements within the room, such as furniture 191-193 (e.g., sofa 191; chair 192; table 193; etc.), pictures or paintings hanging on the wall or televisions or other objects 194 (such as, 194-1 and 194-2), lighting fixtures, etc. The user may also optionally provide a text or auditory identifier to be associated with the collection location, such as "entrance" for collection location 210A or "living room" for collection location 210B, while in other embodiments, the ICA system may automatically generate such an identifier (e.g., by automatically analyzing video and / or other recorded information of the building to perform a corresponding automated determination, such as by using machine learning), or may not use an identifier.

[0041] After the first acquisition location 210A has been appropriately captured (e.g., by a full rotation of the camera device 186 or using one or more wide angle lenses without such a full rotation, and optionally by such a rotation of the mobile computing device 185), the user and / or devices 185 and 186 may proceed to the next acquisition location (such as acquisition location 210B), thereby optionally recording movement data, such as video and / or other data from hardware components (e.g., from one or more IMUs 148, from the imaging system 135, from the distance measurement sensor 136, etc.) by the mobile computing device during movement between the acquisition locations. At the next acquisition location, the camera device 186 may similarly capture one or more target images from the acquisition location, and the mobile computing device 185 may similarly capture additional data at or near the acquisition location. This process may be repeated from some or all of the rooms of the building, and optionally repeated outside the building, as described for acquisition locations 210C to 210M. The video and / or other images captured by the camera device 186 for each capture location are further analyzed to generate a target panoramic image for each of the capture locations 210A to 210M, including in some embodiments stitching together multiple component images to create a panoramic image and / or matching objects and other elements in different images. In addition to generating such panoramic images, in at least some embodiments, further analysis can be performed by a copy of the ILDM system to determine the capture location and optionally the capture orientation of each target image, and by a copy of the MIGM system to determine a floor plan of the building and / or other relevant mapping information of the building (e.g., a determined room shape of the rooms of the building, a set of interconnected linked panoramic images, etc.), for example, to "link" at least some of the panoramas and their capture locations together (where some corresponding lines 215 between example capture locations 210A to 210C are shown for illustration purposes), the MIG The replica of the M system can determine relative position information between pairs of acquisition locations that are visible to each other, store corresponding inter-panoramic links (e.g., links 215-AB, 215-BC, and 215-AC between acquisition locations 210A and 210B, between 210B and 210C, and between 210A and 210C, respectively), and in some embodiments and situations, further link at least some acquisition locations that are not visible to each other (e.g., link 215-BE (not shown) between acquisition locations 210B and 210E; link 215-CM (not shown) between acquisition locations 210C and 210M, etc.).

[0042] about Figure 1A to Figure 1B Various details are provided, but it will be understood that the details provided are non-exclusive examples included for illustrative purposes and that other implementations may be otherwise performed without some or all of such details.

[0043] Figures 2A to 2U Examples of automatically determining a capture position and optionally a capture orientation of a target image captured by a camera device in a building, such as within a floor plan of the building based in part on a determined room shape of the rooms of the building, and examples ... on a determined room shape of the rooms of the building Figure 1B Other examples of target images captured within the discussed building 198 are generated and information is presented on a floor plan of the building.

[0044] In particular, Figure 2A An example image 250a is illustrated, such as that captured by camera device 186 at Figure 1B 1, a non-panoramic stereoscopic image captured from capture location 210B in a northeast direction of a living room of a house 198 (or a northeast-facing subset view of a 360-degree panoramic image taken from the capture location and formatted in a rectilinear manner), in this example further displaying a direction indicator 109a to illustrate the northeast direction in which the image was taken. In the illustrated example, the displayed image includes several visible elements (e.g., lighting fixture 130a), furniture (e.g., chair 192-1), two windows 196-1, and a picture 194-1 hanging on the north wall of the living room. No inter-room passages (e.g., doorways or other wall openings) leading into or out of the living room are visible in the image. However, multiple room boundaries are visible in image 250a, including a horizontal boundary between a visible portion of the north wall of the living room and the living room ceiling and floor, a horizontal boundary between a visible portion of the east wall of the living room and the living room ceiling and floor, and an inter-wall vertical boundary 195-2 between the north wall and the east wall.

[0045] Figure 2B continue Figure 2A , and illustrates the example of the camera device 186 in Figure 1B Additional stereoscopic image 250b is captured from acquisition location 210B in a northwest direction in the living room of house 198, further displaying direction indicator 109b to illustrate the northwest direction in which the image was taken. In this example image, a small portion of one of windows 196-1 continues to be visible along with a portion of window 196-2 and new lighting fixture 130b. In addition, the horizontal and vertical room boundaries are shown with the same Figure 2A A similar approach is visible in image 250b.

[0046] Figure 2C continue FIG. 2A to FIG. 2B , and illustrates the example of the camera device 186 in Figure 1B198, such as a third stereoscopic image 250c captured in a southwest direction from acquisition location 210B, further displaying direction indicator 109c to illustrate the southwest direction in which the image was taken. In this example image, a portion of window 196-2 continues to be visible, as does sofa 191 and visual horizontal and vertical room boundaries. Figure 2A and Figure 2B This example image further illustrates a wall opening for entering / leaving the living room, which in this example is a doorway 190-1 ( Figure 1B It will be appreciated that a variety of other stereoscopic images may be captured and displayed in a similar manner from acquisition location 210B and / or other acquisition locations.

[0047] Figure 2D Explained Figure 1B Other information 255d of the portion of house 198, including the living room and a limited portion of other rooms to the east of the living room. Figure 1B and FIG. 2A to FIG. 2C Discussion, in some embodiments, target panoramic images may be captured at various locations within the interior of a house, such as at locations 210A and 210B in the living room, with corresponding visual content of one or both of such resulting target panoramic images subsequently used to determine the room shape of the living room. Additionally, in at least some embodiments, additional images may be captured, such as where a mobile computing device 185 (not shown) captures a video or one or more other series of continuous or nearly continuous images while moving through the interior of the house. In this example, the Figure 1B The mobile computing device 185 may capture (e.g., if video data is captured) one or more video frame images (or other series of continuous or nearly continuous images) of the surrounding interior of the premises as the mobile computing device 185 moves, examples of such locations include capture locations 240a through 240c, with other information being provided along the path 115 and in particular illustrating a series of locations 215 along the path at which one or more video frame images (or other series of continuous or nearly continuous images) of the surrounding interior of the premises may be captured (e.g., if video data is captured) as the mobile computing device 185 moves. Examples of such locations include capture locations 240a through 240c. Figure 2E to Figure 2J In this example, the locations 215 along the path are shown as being separated by a short distance (e.g., one foot, one inch, a few tenths of an inch, etc.), but it will be appreciated that the video capture may be substantially continuous and, thus, in at least some embodiments, only a subset of such captured video frame images (or other images from a series of continuous or nearly continuous images) may be selected and used for further analysis, such as images separated by a defined distance and / or separated by a defined amount of time between their capture (e.g., one second, a few tenths of a second, a few seconds, etc.) and / or separated based on other criteria.

[0048] Figure 2E to Figure 2J continue FIG. 2A to FIG. 2D 15 and illustrates additional information about a living room and about analyzing 360° image frames from a video captured by a mobile computing device along path 155 as part of determining a type of estimate of the possible shape of the room. Although not illustrated in these figures, similar techniques can be performed for target panoramic images captured by a camera device at two or more of the acquisition locations 210A, 210B, and 210C, whether supplemental analysis Figure 2D Alternatively, the additional image frames shown (e.g., to use the visual data of the target image to generate additional estimates of the possible shape of the room) may be used instead of analyzing Figure 2D Additional image frames are shown. In particular, Figure 2E Information 255e is included, which indicates that the 360° image frame taken from position 240b will share information about various visible 2D features with the 360° image frame taken from position 240a, but for simplicity, only Figure 2E A limited subset of such features for a portion of a living room is illustrated in Figure 2E , example lines of vision 228 are shown from position 240b to various example features in the room, and example lines of similar vision 227 are shown from position 240a to corresponding features, illustrating the degree of difference between perspectives at significantly separated capture locations. Figure 2D A series of images of the location 215 of the living room may provide a variety of information about features of the living room, including information about the associated planes of the features and the normal directions orthogonal to the planes, such as information about FIG. 2F to FIG. 2I Further explanation.

[0049] In particular, Figure 2F information 255f illustrating the northeast portion of the living room visible in the subset of 360° image frames taken from locations 240a and 240b, and Figure 2GInformation 255g of the northwest portion of the living room visible in the other subset of the 360° image frames taken from locations 240a and 240b is illustrated, where various example features in those portions of the living room are visible in both 360° image frames (e.g., corners 195-1 and 195-2, windows 196-1 and 196-2, etc.). As part of automated analysis of the 360° image frames using SLAM and / or MVS and / or SfM techniques, information about planes 286e and 286f corresponding to portions of the north wall of the living room can be determined based on the detected features, and information 287e and 285f about portions of the east and west walls of the living room can similarly be determined based on corresponding features identified in the images. In addition to identifying such planar information of detected features (e.g., each point in the determined sparse 3D point cloud from image analysis), the SLAM and / or MVS and / or SfM techniques may also determine information regarding the likely position and orientation / direction 220 of a subset of images from capture location 240a and the likely position and orientation / direction 222 of a subset of images from capture location 240b (e.g., the position of capture locations 240a and 240b, respectively). Figure 2F Positions 220g and 222g, and optionally Figure 2F The directions 220e and 222e of the image subsets shown; and the capture locations 240a and 240b, respectively. Figure 2G The corresponding positions 220g and 222g in the embodiment, and optionally Figure 2G The directions of the image subsets shown are 220f and 222f). Although Figure 2F and Figure 2G Only features of a portion of the living room are illustrated in , but it will be appreciated that other portions of the 360° image frame corresponding to other portions of the living room may be analyzed in a similar manner to determine possible information about possible planes of various walls of the room and other features of the living room (not shown). Additionally, a similar analysis may be performed between some or all other images at the location 215 selected for use in the living room, resulting in a variety of determined feature planes that may correspond to walls of the room from the various image analyses.

[0050] Figure 2H continue Figures 2A to 2G, and illustrates information 255h from analysis of 360° image frames captured at locations 240a and 240b regarding various determined feature planes that may correspond to the west and north walls of a living room. The plane information shown includes a determined plane 286g near or at the north wall (and therefore a corresponding possible location of the north wall), and a determined plane 285g near or at the west wall (and therefore a corresponding possible location of the west wall). As would be expected, there are many variations in the different determined planes of the north and west walls from the different features detected in the analysis of the two 360° image frames, such as differences in position, angle, and / or length, resulting in uncertainty as to the actual precise position and angle of each of the walls. Although Figure 2H It is not illustrated in , but it will be appreciated that similarly determined feature planes for other walls of the living room will be similarly detected, as well as determined feature planes corresponding to features that are not along walls (e.g., furniture).

[0051] Fig.2I continue FIG. 2A to FIG. 2H example, and illustrates information 255i regarding additional determined characteristic planes that may correspond to the west and north walls of the living room from analysis of various additional 360° image frames selected from additional locations 215 along the path 115 in the living room, as would be expected, in this example, analysis of other images provides an even greater variation of different determined planes for the north and west walls. Fig.2I Also illustrated is additional determined information used to aggregate information about various determined feature planes to identify possible locations 295a and 295b of the west and north walls, such as Figure 2J In particular, Fig.2I Information 291a about normal directions of some of the determined feature planes corresponding to the west wall is illustrated, as well as additional information 290a about those determined feature planes. In an example embodiment, the determined feature planes are clustered to represent assumed wall locations for the west wall, and information about the assumed wall locations are combined to determine possible wall locations 295a, such as by weighting information from various clusters and / or actual determined feature planes. In at least some embodiments, the assumed wall locations and / or normal information are analyzed using machine learning techniques to optionally be optimized by further applying assumptions or other constraints (such as 90° corners, e.g., Figure 2H The resulting possible wall locations may be determined as part of or as a result of a machine learning analysis as illustrated by information 282 of the north wall and / or having a flat wall. A similar analysis may be performed for the north wall using information 290b about corresponding determined characteristic planes and additional information 291b about the resulting normal orthogonal directions of at least some of those determined characteristic planes. Figure 2JResulting possible wall locations 295a and 295b are illustrated which are the west and north walls of the living room, respectively.

[0052] although Fig.2I Although not illustrated in the drawings, it will be appreciated that similarly determined characteristic planes and corresponding standard directions of other walls of the living room will be similarly detected and analyzed to determine their possible locations, resulting in an estimated overall room shape of the living room based on visual data collected in the living room by the mobile computing device 185. Additionally, a similar analysis is performed for each room of the building, thereby providing an estimated room shape for each room. Furthermore, although Figure 2D to Figure 2J Although not described in the specification, in some embodiments, the analysis of visual data captured by the mobile computing device can be supplemented and / or replaced by analysis of depth data (not shown) captured by the mobile computing device in the living room, thereby directly generating an estimated 3D point cloud from depth data representing the walls of the living room and optionally the ceiling and / or floor of the living room and / or directly generating an estimated planar surface from depth data representing the walls of the living room and optionally the ceiling and / or floor of the living room. Figure 2D to Figure 2J Also not described herein, but in at least some embodiments, other room shape estimation operations may be performed using only a single target panoramic image, such as analysis of visual data of the target panoramic image via one or more trained neural networks, as discussed in more detail elsewhere herein.

[0053] Figure 2K continue FIG. 2A to FIG. 2J , and illustrates information 255k about additional information that can be generated from one or more images of a room and used in one or more ways in at least some embodiments. In particular, images (e.g., video frames) captured in a living room of house 198 can be analyzed to determine an estimated 3D shape of the living room, such as from a 3D point cloud of features detected in the video frames (e.g., using SLAM and / or SfM and / or MVS techniques, and optionally further based on IMU data captured by mobile computing device 185). In this example, information 255k reflects an example portion of such a point cloud of the living room, such as in this example, with Figure 2C196-1). In a similar manner to image 250c of FIG. 198 , the estimated 3D shape of the living room may be determined by using depth data captured by mobile computing device 185 in the living room, either in addition to or in lieu of using visual data of one or more images captured by camera device 186 and / or mobile computing device 185 in the living room. Additionally, it will be appreciated that various other walls and other features may be similarly identified in the living room and other rooms of house 198.

[0054] Figure 2L Additional information 255l is illustrated, corresponding to, after determining estimated room shapes for the rooms on the illustrated floor of the house 198, positioning the estimated room shapes of the rooms relative to each other based at least in part on connecting room-to-room corridors between the rooms and matching room shape information between adjacent rooms in this example, which, in at least some embodiments, can be considered constraints regarding the positioning of the rooms and determining optimal or otherwise preferred solutions to those constraints. Figure 2L Examples of such constraints in include making the connecting channel information of adjacent rooms (e.g., in Figure 2E to Figure 2J and / or Figure 2P to Figure 2U 229d and 229e, and rooms 229a and 229b. In other embodiments, various other types of information may be used for room shape locations, such as an exact or approximate size of the overall size of the house (e.g., based on available additional metadata about the building, analysis of images from one or more image acquisition locations outside the building, etc.), whether in addition to or in lieu of the channel-based constraints and / or room shape-based constraints. House exterior information 233 may be further identified and used as a constraint (e.g., based at least in part on automated identification of channels and other features corresponding to the exterior of the building, such as windows), such as to prevent another room from being placed at a location that has been identified as being outside the building.

[0055] Figure 2M to Figure 2-O continue Figures 2A to 2L , and explains how Figures 2A to 2L and Figure 2P to Figure 2U The mapping information generated by the types of analysis discussed in . In particular, Figure 2M An example floor plan 230m is illustrated that may be constructed based on the determined location of the estimated room shape, in this example, the example floor plan 230m includes indications of walls and doorways and windows. In some embodiments, such a floor plan may have other information shown, such as regarding other features automatically detected by the analysis operation and / or subsequently added by one or more users. For example, Figure 2N A modified floor plan 230n is illustrated that includes various types of additional information, such as information that may be automatically identified from analysis operations of visual data from the image and / or depth data and added to the floor plan 230m, including one or more of the following types of information: room labels (e.g., "Living Room" for a living room), room dimensions, visual indications of furniture or appliances or other fixed features, visual indications of the location of additional types of associated and linked information (e.g., panoramic images and / or stereo images captured at a specified capture location that an end user may select for further display; audio annotations and / or sound recordings that an end user may select for further presentation, etc.), visual indicators for doorways and windows, etc. In other embodiments and scenarios, some or all of these types of information may instead be provided by one or more MIGM system operator users and / or ICA system operator users and / or ILDM system operator users. Additionally, when the floor plans 230m and / or 230n are displayed to the end user, one or more user-selectable controls may be added to provide interactive functionality as part of a GUI (graphical user interface) screen 255n, thereby indicating the current floor to be displayed, allowing the end user to select a different floor to be displayed, etc., wherein in this example, corresponding example user-selectable controls 228 are added to the GUI. Additionally, in some embodiments, changes to floors or other floors may also be made directly from the displayed floor plan, such as via selection of corresponding connecting pathways (e.g., stairs leading to different floors), and other visual changes may be made directly from the displayed floor plan by selecting corresponding displayed user-selectable controls (e.g., a control to select a particular image corresponding to a particular location, and receive display of that image, either in place of or in addition to a previous display of the floor plan from which the image was selected). In other embodiments, information for some or all of the different floors may be displayed simultaneously, such as displaying separate sub-floor plans for separate floors or instead integrating all room and floor room connection information into a single floor plan displayed together at once. It will be appreciated that in some embodiments various other types of information may be added, that in some embodiments some of the illustrated types of information may not be provided, and that in other embodiments visual indications and user selections of links and associated information may be displayed and selected in other manners.

[0056] Figure 2-O continue FIG. 2A to FIG. 2N and illustrates that it is possible to use the disclosed and displayed herein (e.g., in a manner similar to Figure 2N GUI of the present invention), in this example, the additional information 265o is a 2.5D or 3D model floor plan of the house. Such a model 265o may be additional mapping-related information generated based on the floor plan 230m and / or 230n, where additional information about height is shown to illustrate the visual location of features such as windows and doors in the wall. Although Figure 2-O Not illustrated, but in some embodiments, additional information can be added to the displayed walls, such as from images taken during video capture (e.g., rendering and illustrating actual paintings, wallpaper, or other surfaces from a house on the rendered model 265), and / or can be used in other ways to add specified colors, textures, or other visual information to walls and / or other surfaces.

[0057] Figure 2P to Figure 2U continue Figure 2A to Figure 2-O An example of Figure 2P Explains the Figure 1B Other information 230p of the living room of the house 198, and information about the target image captured by the camera device, the capture location information of the target image will be determined by the automated operation of the ILDM system. In this example, the target image capture corresponds to Figure 1B The collection location 210B shown, and in particular at Figure 2P The capture position 241a is shown as being taken by the camera device 186 at a starting orientation direction 243a (eg, if the camera device was initially pointed in that orientation direction and then rotated 360° at the capture position 241a). Figure 2P Also illustrated is a nearby location 241b where the associated mobile computing device 185 captures yet another image at an orientation direction 243b, such as at a time similar to the capture time of the target image. A previous target image may have also been captured at a capture location 210A in the living room at a previous time, where its capture location information (not shown) was previously determined. Additionally, Figure 2PAdditional movement information 244 of the mobile computing device is illustrated, which in this example corresponds to a user (not shown) placing the camera device 186 in a collection location 241a (e.g., on a tripod or other mount on which the camera device will rotate 360°, on a fixed tripod or other mount if the camera device 186 includes one or more wide-angle lenses that allow for simultaneous capture of 360° horizontal visual coverage), etc.), and then moving to a location where the camera device 186 is not visible to the user at the collection location 241a (e.g., down a hallway), and then returning to the collection location 241a to resume the camera device 186. In other embodiments, movement data 244 may not be collected for this purpose (e.g., if the target image is captured by the user holding the camera device and rotating the user's body in a 360° circle and thus not leaving the capture location area during capture of the target image), but as discussed in more detail elsewhere in this document, if such movement of the mobile computing device occurs, the movement information 244 can be tracked by the mobile computing device (e.g., tracked by one or more IMU sensors, visual data from one or more image sensors, etc.) and used as at least part of determining the capture location 241a of the target image.

[0058] Figure 2Q and Figure 2R continue Figure 2P An example of Figure 2Q 250q is illustrated as being captured by the camera device 186 at the capture position 241a. In this example, the target image is a 360° panoramic image (in Figure 2Q spherical format to simultaneously show the full 360° horizontal visual content of the target image). Figure 2R 198. Visual data of yet another image captured by the associated mobile computing device 185 in the living room of the house 198 is illustrated, in this example, the image being a 180° panoramic image 255r (at Figure 2R ), but in other cases the image may be a non-panoramic stereo image (e.g., Figure 2R250d, in this example, image 250d is shown as a subset of image 255r). As shown in the example of yet another image 255r, the visual data of the yet another image may include some or all of camera device 186 and may also be used as part of determining acquisition position information for the target image, such as identifying one or more optional markings 257 on camera device 186 in the visual image of the yet another image and / or identifying the camera device itself (e.g., identifying its shape) in the visual data of the yet another image. For example, automated determination of acquisition position information for the target image may include identifying an offset distance 238a between the positions of the acquired image 255r and / or identifying an offset orientation 238b between an orientation 243a at which the target image was acquired and an additional orientation 243b at which the yet another image was acquired, wherein the offset information is used to determine relative acquisition position information of camera device 186 for the target image relative to a mobile computing device for the yet another image. Alternatively, if the yet another image is instead example image 250d, then in the example, such visual information about camera device 186 may not be available.

[0059] After being supplied with the target image and further captured data, the ILDM system can perform various automated operations to determine the capture location of the target image using the visual content of the target image from the camera device and additional data from the mobile computing device, including, in some cases, determining that the target image was captured in the living room (rather than in another room in the building 198, or in some cases, in other rooms in other buildings) and determining the specific capture location and orientation of the target image within the living room, wherein such determined capture location of the target image is subsequently superimposed on a displayed partial or all of a floor plan of the building including at least the determined room shape of the living room.

[0060] In particular, Figure 2S continue Figure 2P to Figure 2R , and includes information 255s illustrating one or more types of estimated room shape information 237 of a living room based on an analysis of visual data of a target image captured by a camera device by an ILDM system. In particular, in this example, the estimated room shape information 237 includes information 237a about structural wall elements identified in the visual data (e.g., wireframe lines of a room outline), and / or information 237b showing a 2D estimated room shape of the living room, and / or information 237c showing a 3D estimated room shape of the living room. In this example, the locations of structural wall elements (such as windows and doorways and other wall openings) can be automatically identified and shown on the room shape information 237b and / or 237c. As discussed in more detail elsewhere herein, estimating the room shape of the living room from the visual data of the target image can be performed in various ways. Figure 2SOne or more additional types of estimated room shape information 239 for the living room are also illustrated based on the ILDM system's analysis of additional data captured by the mobile computing device. In particular, in this example, the estimated room shape information 239 includes information 239a about structural wall elements identified in the visual data, and / or information 239b showing a 2D estimated room shape of the living room, and / or information 239c showing a 3D estimated room shape of the living room. In this example, the estimated room shape information 239a and 239b include solid lines to correspond to the in-line shape of the living room. Figure 2R 255r, and the estimated room shape information 239c only includes information corresponding to that portion of the living room. However, if other additional images (not shown) have visual data for other portions of the living room, then the combination of visual images from multiple other images may also include the information shown in dashed lines in the estimated room shape information 239a and 239b, and can be used to add the additional information to the estimated room shape information 239c. It will be appreciated that the automated determination of the estimated room shape information 237 and 239 may not produce a perfect room shape, so that there are differences between the estimated room shape information 237 and 239.

[0061] Figure 2T continue Figure 2P to Figure 2S Examples and included in the comparison Figure 2SThe information 255t of some of the further automated operations of the ILDM system is illustrated in the multiple estimated room shapes 237 and 239. As discussed in more detail elsewhere in this document, various matching criteria can be used to determine the fit between two room shapes. As an example, the distance between one or more (e.g., all) matching corners of the estimated room shape and the determined room shape can be measured, such as for the two corners in the upper left (northwest) corner, shown as 234b, where the smaller the distance, the better the match (and in some embodiments, the higher the matching score). Although other corners are not shown in this example, they can be similarly measured and evaluated for the distance of the matching corners. As another example, the distance between one or more (e.g., all) corresponding wireframe structure lines of the estimated room shape and the determined room shape can be measured, such as for the two wireframes corresponding to the boundary of the north wall to the floor, shown as 234c, where the smaller the distance, the better the match (and in some embodiments, the higher the matching score), although in this example, it is not shown for other structure lines, but the distance between their corresponding lines can be similarly measured and evaluated. As yet another example, the distance between one or more (e.g., all) structural wall elements identified in the estimated room shape may be measured (e.g., using the corresponding 3D bounding boxes of those structural wall element objects), such as for two doorways at the lower left side of the room, shown as 234a, and where smaller distances reflect better matches (and in some embodiments, higher match scores), although in other embodiments the entire bounding box area may be compared (e.g., using intersection and uniform distance measures). Although not shown in this example for other structural element objects, the distances between their corresponding bounding box portions may be similarly measured and evaluated. As another example, the distance between one or more (e.g., all) corresponding walls of multiple room shapes may be measured, such as for two walls on the top side of the room shape, shown as 234d, where smaller distances reflect better matches (and in some embodiments, higher match scores). Although not shown in this example for other walls, the distances between their corresponding walls may be similarly measured and evaluated. Various other criteria may be accessed and used, either in addition to or in place of some or all of the criteria described, as discussed in more detail elsewhere herein. Various distance offsets between multiple room shapes may be combined and used to determine where the acquisition position 241a and optionally the acquisition orientation 243a of the target image within the room shape information occurs. Figure 2TOnly two degrees of freedom are shown for each of the acquisition position and orientation, but the actual determined acquisition position and orientation information can be determined with three degrees of freedom for each of the acquisition position and orientation (e.g., the height of the acquisition position above the floor, and the upward or downward angle of the acquisition position height from the acquisition position). In addition, once the acquisition position information is determined for such a target image, it can be displayed on an updated version of the floor plan of the building, such as for Figure 2U The information 255u is described in the updated floor plans 230u and 265u.

[0062] Already referenced Figures 2A to 2U Various details are provided, but it will be understood that the details provided are non-exclusive examples included for illustrative purposes and that other implementations may be otherwise performed without some or all of such details.

[0063] As a non-exclusive example implementation, automated operations of the ILDM system may include the following operations: determining a capture location (e.g., capture position and optionally capture orientation) of a target panoramic image captured by a camera device by using visual data of the target panoramic image and additional data captured by an accompanying mobile computing device (in this example, a mobile smart phone computing device or "phone"). The various operations may, for example, include one or more of the following:

[0064] - Visual image recognition and marker tracking. One or more markers (e.g. a pair of QR-type codes printed in color at a size such as 4cm×4cm) are attached to an external camera (e.g. one on each of the two sides of the camera); or if the camera itself has a unique visual texture of suitable size, contrast and distinctiveness (such as a brand logo), these may be used instead or in addition. After the camera is placed, the phone recognizes and tracks these images using its own camera and IMU data to determine the pose of the camera relative to the phone using SLAM technology. This image recognition may involve the user manually "scanning" the markers by waving the phone towards them within a set range of distances, where the user then triggers a panoramic capture within a set time, then moves the panoramic camera again, and because the pose of the phone continues to be tracked in a consistent coordinate system, the pose of the markers and therefore the camera can be determined relative to the phone at the time of capture.

[0065] -Object recognition. Similar to 2D marker recognition and tracking, this approach instead uses 3D shape recognition to recognize and track the shape of the camera itself (or the camera plus a tripod or other support).

[0066] -Room shape matching. Preferably for the room in which the panorama is captured, an approximate model of the local geometry of the house in the immediate vicinity of the panorama is reconstructed. In some cases, this room shape estimation automation operation may also involve obtaining and using human annotation information, while in other cases, the room shape estimate is automatically extracted without any such human annotation information. At the same time, additional data of the phone (e.g., visual data and / or IMU data) can be used to reconstruct the three-dimensional geometry, and this additional data can be particularly advantageous in identifying vertical and horizontal planes. Both processes can be enhanced by identifying well-known objects (such as windows and doors), and can optionally be enhanced by identifying additional object types in the room (e.g., furniture). In this shape reconstruction, the position of the phone or camera can be determined by the characteristics of the reconstruction process. By making these two estimates of the room shape (and possibly other objects) and finding correspondences between them, the relative pose of the two origins (camera and phone) can be determined. Even if the initial estimate of the room shape has some uncertainty, the unknown quantity (relative pose) consists of only six degrees of freedom, resulting in an additional specified problem, where a large number of correspondences can help reduce the uncertainty of the overall solution through least squares fitting or similar techniques.

[0067] -Pose estimation from local visual feature matching. Unique visual features can be identified from one image and located in another, even when they are rotated, scaled, projected differently in the second image, or have different exposure levels or color balance. This feature matching technique can be applied to panoramic images taken from a 360 camera and to one or more frames captured from the phone's camera. Given that the system only solves six unknowns for relative pose, techniques can be used to maximize the number of internal matches to obtain the best pose estimate (e.g., RANSAC or random sample consistency). Since the pose of the phone as it captures each frame from its built-in camera can be determined from its visual data and / or IMU data, this allows the pose of the external camera to be determined similarly to the room shape matching method described above.

[0068] - Depth / point cloud matching. The phone may be able to directly measure or infer depth information through techniques such as stereo, structured light, time-of-flight sensors (e.g., LiDAR sensors). Additionally, for external cameras, depth information may be directly inferred from a still RGB image obtained from the camera. Both depth maps may be used to generate a point cloud, and then the best correspondence between the two point clouds may be found, providing the relative pose. This technique may also be combined with the respective RGB information, improving the information available to perform the match.

[0069] - Motion analysis. To capture a panorama from an external camera, a user may perform an activity including: placing a 360 camera at a capture location; walking to an area not visible to the 360 ​​camera (e.g., walking to another room); pressing a button on the phone or waiting for a timer to elapse to trigger capture; and returning to pick up the 360 ​​camera. During this movement, the user's path may be captured by the additional data it collects, and the point in the path where the camera is placed may be approximated (e.g., by using pattern recognition techniques). Even if there is some uncertainty in the position and / or orientation of the camera according to the analysis, some of the other techniques are still beneficial if the initial estimate is used as a seed to find a newer (e.g., higher accuracy) solution.

[0070] - Visual matching between panoramas for coordinate system fusion. The pose data determined for the phone from its additional data may be discontinuous, such as due to gaps in the acquisition of the additional data, and if so, a pose path that attempts to insert through these gaps can be used by "relocalization" (finding correspondences between the old camera and depth data available before the gap and the new camera and depth data available after the gap). If this relocalization fails, subsequent poses and geometry data will be specified in a new, independent coordinate system that is independent of the coordinate system used before the gap. These independent coordinate systems (each corresponding to a local world map) will prevent a single global positioning of all panoramas. If these coordinate systems are fused / combined (aligning them like pieces of a puzzle), multiple independent coordinate systems can be removed or reduced. One way to do this is to determine a set of relative positioning of a pair of panoramas, where each panorama in the pair is in a different coordinate system. This can in turn be done using techniques related to those described above, such as finding feature correspondences between two panoramas rather than between a panorama and a phone camera frame. If the number of panoramas is significantly higher than the number of independent coordinate systems, only a few cross coordinate system pairs can be used to fuse the coordinate systems.

[0071] - Rigid mount. This information can be used in limited implementations where both the phone and external camera are attached to the same tripod or other mount in such a way that the position and orientation of the external camera relative to the phone is always the same, but such an arrangement generally provides significantly reduced usability, including that the phone is often not visible in the final panorama.

[0072] The various operations have various advantages and disadvantages, for example, room shape matching may not work fully in crowded or unusually shaped rooms; and motion analysis may introduce positioning uncertainty and fail to provide camera orientation. Benefits can be achieved by using multiple positioning techniques together, including using different techniques in different situations, and using the results of some techniques as initial estimates updated by other techniques (e.g., using motion pattern matching and / or camera marker recognition as initial estimates used by optimization-based techniques such as depth / point cloud matching and RGB feature matching). In addition, multiple candidate results and / or confidence information from each of the multiple techniques can be used to combine the results from the multiple techniques (e.g., discarding the results with the lowest confidence from one or more techniques; combining the results using statistical analysis, such as discarding outliers or selecting median values; etc.).

[0073] After performing the automated acquisition position determination operation for each of the multiple target panoramic images acquired for the building, and if applicable coordinate system mapping has been identified to allow the multiple coordinate systems to be combined into a single coordinate system, the entire set of panoramic positioning (6 degrees of freedom each) and coordinate system mapping (up to 5 or 6 degrees of freedom per pair) can be combined into a small set of global systems (one for each set of disconnected coordinate systems). Additional constraints or objective functions can also be applied based on knowledge or assumptions about the overall floor plan geometry (such as room non-intersection or door matching), and optimization-based techniques can again be employed to simultaneously optimize each of these nearly global systems, thereby providing an improved set of global results. This global optimization activity and the resulting information can be updated each time a new target panoramic image is added.

[0074] Once automated acquisition position information is determined for such a target panoramic image, the information may be used in a variety of ways, such as one or more of the following:

[0075] -Displays an approximate floor plan and determines which areas of the house are well covered by the panorama and which other areas to capture next. This allows the user to capture higher quality data and avoid costly mistakes that currently require returning to the site to capture additional images or even require a new tour.

[0076] - Determining a set of panoramas with directional links between them so as to use them for a virtual tour. Such a virtual tour provides various benefits, including allowing a user to preview the tour during the capture of a target panoramic image.

[0077] - The user’s path is further analyzed and fused with collected images to improve detection of stairs and doorways, thereby enhancing floor plan generation and tour generation.

[0078] - Combine geometry information obtained via SLAM on the phone with objects detected in the panoramic image to form a rich understanding of light fixtures, furniture, and other items of interest within the house. This information can be used during visualization of the space for the end user and for additional automated analysis performed.

[0079] - Generating (and optionally using) an initial estimate of the acquisition location of the target panoramic image on the phone, and also using the initial estimate as input to other techniques for updating the initial estimate (e.g., after completing capturing the building, and on a server computing system during later construction of a 3D model of the building).

[0080] As another non-exclusive example implementation, the automated operation of the ILDM system may include the following actions. Start with one or more target images having RGB visual data (but no separate depth data), optionally with one or more other acquisition metadata of the target images, which may include image capture timestamps, camera IMU signals, camera SLAM-based tracking data, and / or image room tags (e.g., supplied by a user who captured the target image of its enclosed room). Additionally, start with a 2D or 3D floor plan that uses a vector format and optionally has existing images at specified locations and in different image projections from any of the target images, and / or has additional images in different image projections from any of the target images with optional other acquisition metadata of the same type.

[0081] Automated operations can include pre-processing the target image and optionally additional images and optionally existing images to resolve camera intrinsics and extrinsics (if necessary), such as detecting image vanishing lines and vanishing points, extracting (if the image is stereo) camera focal length and field of view angle, resolving camera tilt, roll, and shake angles relative to vanishing lines present in the image, and re-projecting the image into spherical space (with a new camera pose tilted relative to the floor plane).

[0082] The automated operations may also include generating geometry predictions for each target image and optionally each additional image and optionally each existing image, including the following: estimating the room shape geometry of the indoor structure where the camera is located (e.g., using a convolutional neural network based room shape estimator, such as HorizonNet and DuLaNet, to approximate the room shape geometry to a 3D shape with uniform room height, where the camera can be at the origin of the shape); optionally using an image structure wireframe estimator (e.g., LCNN) to predict image structure lines and projecting these lines in the image 3D space as room corner candidates; using an object detection algorithm on the image to generate a 2D object bounding box with a label and object image descriptor and then ray-casting the 2D image bounding box onto the previously estimated 3D room shape and generating a footprint of the 3D object to represent the spatial information of the object, and using a 3D bounding box generation algorithm; optionally generating an image embedding vector (e.g., using a deep neural network model) for later use in comparing image content similarity and image overlap; and optionally labeling the image with one or more room types (e.g., bedroom, kitchen, etc.).

[0083] The automated operations may also include generating image-to-image relationships between each target image and one or more additional images and / or existing images, including the following: optionally using a feature-based image matching algorithm such as SfM between image pairs to solve for image angle connections or pairwise image position information (e.g., which orientation in image A connects to which orientation in image B); and optionally using a deep learning-based image co-visibility algorithm between image pairs to determine image content similarity (e.g., for later use in the hypothesis that images sharing high co-visibility scores have a high chance of being spatially closer to each other).

[0084] The automated operation may also include retrieving a set of room shape candidates on which to attempt to position each target image in order to determine an accurate acquisition location for the target image, the room shape candidates may be obtained from existing room shapes associated with the floor plan and / or room shapes estimated for a set of spatially related additional images. Various heuristics may be used to generate binary relationships between a pair of target images and additional images or between a target image and an area in an existing floor plan, including the following: using similarities / overlaps between room type labels for the target image and the paired images / areas (if available, such as created by an automated image classification algorithm and / or the photographer and / or subsequent annotators) to aggregate a list of preferred candidate room shapes; using temporal relationships between images (if image capture timestamp metadata is available) to retrieve a set of temporally related additional images; using feature matching-based image alignment algorithms to generate common relationships for pairs or groups of images (e.g., image relative angles or binary image common relationships); using neural network-based image comparison algorithms to generate pairwise image-to-image common relationships; using IMU metadata collected during the image capture process (if available) to give image angular connections; and using SLAM-based camera tracking algorithms (if SLAM data is available) to produce image spatial relationships.

[0085] The automated operation may also include performing geometric shape matching with one or more candidate room shapes for each target image to match the estimated room shape of the target image with a corresponding determined room shape of a room on a floor plan or with a corresponding estimated room shape of an additional image, and localizing the target image to a single room shape (e.g., to produce one or more camera pose acquisition locations of the target image, optionally along with a confidence score for each camera pose). The automated operation generally includes the following operations: proposing a number of shape matching options (which are based on the target image camera pose in the candidate room shape space); calculating a score for each proposed camera pose (proposed shape matching location); selecting the camera pose with the highest score or using a threshold to pick multiple camera poses; and refining one or more selected camera poses.

[0086] Proposing various shape matching options may include assuming that the 2 room shapes have the same scale if the 2 room shapes are captured by the same camera at the same height (such as for one or more target images and one or more additional images captured simultaneously during the same time period). The corners of the room shapes are used to generate a set of corner capture options (alternative shape matches) with different shape orientations between the existing room shape of the target image and the candidate room shape. The shape orientation is generated by capturing the horizontal vanishing angle of the target image to the vanishing angle of the paired additional or existing image or candidate room shape. Therefore, if there are M predicted room corners in the target image, N room corners in the candidate room shape, and there are 4 vanishing directions from the target image and the paired additional or existing image, then M*N*4 camera poses are proposed for the target image. When the 2 images are captured at inconsistent camera heights, camera poses can be proposed by selecting 2 control corners from each shape and using them to generate the proposed scale and xyz, where the vanishing angle alignment is used to correct the proposed camera angles.

[0087] Calculating a score for each proposed camera pose (proposed shape matching position) may include combining multiple individual scores given the proposed camera pose (e.g., taking a weighted sum of each individual score, extracting descriptors from each of these items and using a machine learning model to generate a final score, etc.). The individual scores may include one or more of the following: a corner reprojection score, in which the candidate room shape is reprojected into the target image space, the projected room corners from the candidate room shape are compared to the room corners of the existing room shape from the original target image, and each target room corner is matched to its nearest candidate room shape corner, and the distance of each matched corner pair and the number of matches are used to generate a corner reprojection score (e.g., the closer the match, the higher the score); a wireframe structure line reprojection score, in which the structure lines of the candidate room shape are reprojected into the target image space, the projected structure lines from the candidate room shape are compared to the structure lines from the target image estimated room shape, and each target image structure line is matched to its nearest candidate room shape structure line, and the distance of each matched structure line pair and the number of matches are used to generate a wireframe structure line reprojection score (e.g., the closer the match, the higher the score); a structural wall element object reprojection score, in which the 3D object bounding box of the candidate room shape from the candidate room shape is reprojected into the target image estimated room space, and the projected structure lines from the candidate room shape are compared to the structure lines from the target image estimated room shape, and each target image structure line is matched to its nearest candidate room shape structure line, and the distance of each matched structure line pair is used to generate a wireframe structure line reprojection score (e.g., the closer the match, the higher the score); a shape-based projected object bounding box to an object bounding box of an estimated room shape from a target image, and each target image object bounding box is matched to its nearest candidate room shape object bounding box, using intersection-over-union based distances and consistency of object type labels for each matched object bounding box pair; an image angle score, wherein departure / landing angles starting from the target image to an additional / existing image are generated, wherein separate departure / landing angles are also generated for each pair of images using different techniques (e.g., SfM, convolutional neural networks, etc.), and wherein a score is calculated by comparing these 2 sets of angles (e.g., where the greater the difference, the more penalty is placed in the score); an image content matching score, wherein image content similarity is generated for a given pair of images (e.g., using a convolutional neural network); and a shape-based boundary intersection score, wherein structural walls of the candidate room shape are re-projected in the 3D space of the target image, and the mismatch between the structural walls of the projected room shape and the target image estimated room shape is used to evaluate the proposed camera pose.

[0088] Refining one or more selected camera poses may include using an initial camera pose of a target image from a previous operation (e.g., using corner point matching) and refining the camera pose using one or a combination of steps. The steps may include one or more of the following: performing alignment using corner inliers, wherein all matching pairs from a previous corner matching operation within a specific re-projected image distance are filtered using a distance threshold (where the resulting corner pairs are referred to as corner inliers), and finding the best camera position xyz using a weighted least squares method, wherein confidence scores of predicted corners of an estimated room shape from the target image (e.g., as generated by a neural network model) are used as weights in a weighted least squares regression method to generate a camera position that is more accurate than a previous camera pose; performing alignment using line matching of wireframe structural line predictions of the target image and the candidate room shape (e.g., between horizontal lines on a floor), such as by a distance threshold used to filter all matching lines from a previous line matching operation within a specific re-projected image distance (where the resulting line pairs are referred to as line normals), and using a weighted least squares method to find an optimal camera position xyz, where confidence scores of predicted structure lines from an estimated room shape from a target image (e.g., as generated by a neural network model) are used as weights in a weighted least squares regression method to generate a camera position that is more accurate than a previous camera pose; and using image normal prediction to perform a differentiable rendering optimization method, where the camera pose is optimized for a low-cost function value by rendering pixel-level surface normal information of a candidate room shape in a target image space starting from an initial camera pose guess, comparing the rendered surface normals with surface normals estimated from the target image in its image space (e.g., using a neural network-based method such as Taskonomy), and calculating a cost value so as to iteratively optimize the camera pose until the cost value reaches a local minimum.

[0089] Various details have been provided above with respect to these example non-exclusive embodiments, but it will be understood that the details provided are included for illustrative purposes and other embodiments may be otherwise performed without some or all of such details.

[0090] Figure 3300, and one or more server computing systems 380 that perform an implementation of an ILDM system 340, and one or more server computing systems 380 that perform an implementation of an ICA system 387 and a MIGM system 388, which may be implemented using a plurality of hardware components that form electronic circuits adapted and configured to perform at least some of the techniques described herein when operating in conjunction. In the illustrated implementation, each server computing system 300 includes one or more hardware central processing units ("CPUs") or other hardware processors 305, various input / output ("I / O") components 310, storage devices 320, and memory 330, wherein the illustrated I / O components include a display 311, a network connection 312, a computer-readable medium drive 313, and other I / O devices 315 (e.g., a keyboard, a mouse or other pointing device, a microphone, a speaker, a GPS receiver, etc.). Each server computing system 380 may include hardware components similar to those of server computing system 300 , including one or more hardware CPU processors 381 , various I / O components 382 , storage devices 385 , and memory 386 , although some details of server 300 are omitted in server 380 for the sake of brevity.

[0091] The server computing system 300 and the executed ILDM system 340 can communicate via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.) with other computing systems and devices, such as the following: a user client computing device 390 (e.g., for viewing floor plans, associated images and / or other related information); an ICA and MIGM server computing system 380; one or more mobile computing devices 360; optionally one or more camera devices 375; optionally other navigable devices 395 that receive and use floor plans and determined image acquisition locations and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices); and optionally other computing systems not shown (e.g., for storing and providing additional information related to buildings; for capturing building interior data; for storing and providing information to client computing devices, such as additional supplemental information associated with images and the buildings or other surroundings they cover; etc.). In some embodiments, some or all of the one or more camera devices 375 can communicate directly (e.g., wirelessly and / or via a cable or other physical connection, and optionally in a peer-to-peer manner) with one or more associated mobile computing devices 360 in their vicinity (e.g., to transmit captured target images, receive instructions to initiate target image acquisition, etc.), whether in addition to or in lieu of performing communications via the network 399.

[0092] In the illustrated embodiment, an embodiment of the ILDM system 340 is executed in the memory 330 to perform at least some of the described techniques, such as using the processor 305 to execute software instructions of the system 340 in a manner that configures the processor 305 and the computing system 300 to perform automated operations that implement those described techniques. The illustrated embodiment of the ILDM system may include one or more components (not shown) to each perform a portion of the functionality of the ILDM system, and the memory may further optionally execute one or more other programs 335, as a specific example, in at least some embodiments, a copy of the ICA and / or MIGM system may be executed as one of the other programs 335, such as in place of or in addition to the ICA system 387 and the MIGM system 388 on the server computing system 380. The ILDM system 340 may also store and / or retrieve various types of data on the storage device 320 (e.g., in one or more databases or other data structures) during its operation, such as information 321 regarding a target image (e.g., acquired by one or more camera devices 375) for which an acquisition position is to be determined and associated information 325 regarding such determined acquisition position and, optionally, acquisition orientation; information 323 regarding an estimated 3D room shape and wall element information for a target image (e.g., generated by the ILDM system during its automated operation); various types of floor plan information and other building mapping information 326 (e.g., generation and preservation of a room enclosing the target image); and information 327 regarding the room's orientation and orientation. generated and saved 3D room shapes; generated and saved 2D floor plans having 2D room shapes and locations of wall elements and other elements on these floor plans, and optionally additional information (such as building and room dimensions for use with the associated floor plans, existing images with specified locations, annotation information, etc.); generated and saved 2.5D and / or 3D model floor plans that are similar to 2D floor plans but also include height information and 3D room shapes; etc.); user information 328 about a user of a client computing device 390 and / or an operator user of a mobile device 360 ​​interacting with the ILDM system; and optionally various other types of additional information 329.The ICA system 387 and / or MIGM system 388 may similarly store and / or retrieve various types of data on the storage device 385 (e.g., in one or more databases or other data structures) during their operation and provide some or all of this information to the ILDM system 340 for use (whether in a push or pull manner), such as images 393 (e.g., 360° target panoramic images captured by one or more camera devices 375 and transmitted to the server computing system 380 by those camera devices and / or by one or more intermediate associated mobile computing devices 360); directional link information 396 between target images generated by the IGM system to represent corresponding buildings; derived floor plan information and optionally other building mapping information 391 (e.g., similar or identical to information 326) generated by the MIGM system for such buildings; additional information generated by the MIGM system as part of generating the floor plan (such as determined room shapes 392 and image location information 394); and optionally various types of additional information 397 (e.g., various analytical information related to the presentation or other use of one or more building interiors or other environments captured by the ICA system).

[0093] Some or all of the user client computing device 390 (e.g., mobile device), mobile computing device 360, camera device 375, other navigable device 395, and other computing systems may similarly include some or all of the same types of components described for server computing systems 300 and 380. As a non-limiting example, the mobile computing device 360 ​​is each shown to include one or more hardware CPUs 361, I / O components 362, storage devices 365, optionally imaging systems 364, optionally IMU hardware sensors 369, optionally depth sensors 363, and memory 367, wherein one or both of a browser and one or more client applications 368 (e.g., applications specific to the ILDM system and / or ICA system) are optionally executed within the memory 367, such as participating in communications with the ILDM system 340, the ICA system 387, the associated camera device 375, and / or other computing systems. Although specific components are not described for other navigable devices 395 or client computing systems 390, it will be appreciated that they may include similar and / or additional components.

[0094] It will also be appreciated that computing systems 300 and 380 and camera device 375 and Figure 3Other systems and devices included in the invention are merely illustrative and are not intended to limit the scope of the invention. The system and / or device may be changed to include multiple interactive computing systems or devices, and may be connected to other devices that are not specifically described, including connecting via Bluetooth communication or other direct communication, through one or more networks (such as, the Internet), via the Web or via one or more dedicated networks (e.g., mobile communication networks, etc.). More generally, the device or other computing system may include any combination of hardware that can interact and perform the functions of the described types when programmed or otherwise configured with specific software instructions and / or data structures, including but not limited to desktop computers or other computers (e.g., tablet computers, tablet computers, etc.), database servers, network storage devices and other network devices, smart phones and other cellular phones, consumer electronic devices, wearable devices, digital music player devices, handheld game devices, PDAs, wireless phones, Internet appliances, camera devices and accessories, and various other consumer products including appropriate communication capabilities. In addition, in some embodiments, the functions provided by the ILDM system 340 shown may be distributed in various components, some of the described functions of the ILDM system 340 may not be provided, and / or other additional functions may be provided.

[0095] It will also be appreciated that, although various entries are described as being stored in memory or on a storage device when in use, for the purposes of memory management and data integrity, these entries or portions thereof may be transferred between memory and other storage devices. Alternatively, in other embodiments, some or all of the software components and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Thus, in some embodiments, when configured by one or more software programs (e.g., an ILDM system 340 executed on a server computing system 300) and / or data structures, some or all of the described techniques may be performed by hardware components including one or more processors and / or memories and / or storage devices, such as by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures, and such as by executing algorithms as described in the flow charts and other disclosures herein. Furthermore, in some embodiments, some or all of the systems and / or components may be implemented or provided in other ways, such as by being comprised of one or more components that are partially or completely implemented in firmware and / or hardware (e.g., rather than as components that are implemented in whole or in part by software instructions that configure a particular CPU or other processor), including, but not limited to, one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium (such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), network storage device, or portable media article (e.g., DVD disk, CD disk, optical disk, flash memory device, etc.) to be read by an appropriate drive or via an appropriate connection. In some embodiments, the systems, components, and data structures may also be transmitted via a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on a variety of computer-readable transmission media including wireless-based and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multi-channel analog signal, or as multiple discrete digital packets or frames). In other embodiments, such a computer program product may also take other forms. Therefore, other computer system configurations may be utilized to practice embodiments of the present disclosure.

[0096] Figure 4 An example flow chart illustrating an implementation of an ICA system routine 400 is shown. The routine may be implemented by, for example, Figure 1A ICA system 160, Figure 3 The ICA system 387 and / or the ICA system execution as otherwise described herein, such as to acquire 360° panoramic images of a target and / or other images within a building or other structure (e.g., for subsequent generation of related floor plans and / or other mapping information, such as generated by an embodiment of the MIGM system routine, wherein the FIG. 5A to FIG. 5B An example of such a routine is described; for subsequently determining the acquisition position and optionally the acquisition orientation of the target image, such as determined by an embodiment of the ILDM system routine, wherein FIG. 6A to FIG. 6B 4. An example of such a routine is described; etc.). Although portions of the example routine 400 are discussed with respect to capturing a particular type of image at a particular location, it will be appreciated that this or similar routines may be used to capture video or other data (e.g., audio) and / or other types of images that are not panoramic, either in lieu of or in addition to such images. Additionally, although the illustrated embodiment captures and uses information from the interior of a target building, it will be appreciated that other embodiments may perform similar techniques for other types of data, including for non-building structures and / or for information on the exteriors of one or more target buildings of interest. Furthermore, some or all of the routines may be performed on a mobile device used by a user to participate in capturing image information and / or related additional data, and / or by a system remote from such a mobile device.

[0097] The illustrated implementation of the routine begins at block 405, where an instruction or information is received. At block 410, the routine determines whether the received instruction or information indicates to collect data representing the interior of a building, and if not, continues to block 490. Otherwise, the routine proceeds to block 412 to receive an instruction (e.g., from a user of a mobile computing device associated with one or more camera devices) to start an image collection process at a first collection location. After block 412, the routine proceeds to block 415 to perform a collection location image collection activity to collect at least one 360° panoramic image (and optionally one or more additional images and / or other additional data, such as from an IMU sensor and / or a depth sensor) by at least one camera device for a collection location in the interior of a target building of interest, such as to provide horizontal coverage of at least 360° around a vertical axis. The routine may also optionally obtain annotations and / or other information about the collection location and / or surroundings from the user, such as for later use in presenting information about the collection location and / or surroundings.

[0098] After completing box 415, the routine continues to box 417 to determine whether determination of acquisition position information (e.g., one or more acquisitions and optionally acquisition orientations) of one or more target images acquired in box 415 is to be performed at the current time, and if so, the routine continues to box 419 to perform automated operations of the image position determination manager routine to determine the acquisition position information of the target image. FIG. 6A to FIG. 6B An example of such an image location determination manager routine is shown. After block 419, the routine continues to block 421 to optionally display information about the determined acquisition location information for one or more target images on one or more of the acquisition devices (e.g., on a mobile computing device), such as, in some embodiments and scenarios, displaying a local floor map of a building on the mobile computing device for the rooms in which the images were acquired (e.g., based at least in part on displaying the determined 2D or 3D room shape information for those rooms), as well as information about the determined target image acquisition location information.

[0099] After block 421, or if it is instead determined in block 417 that acquisition location information for one or more target images acquired in block 415 is not determined at the current event, the routine continues to block 425 to determine whether there are more acquisition locations to acquire images, such as based on corresponding information provided by a user of the mobile computing device. If so, the routine continues to block 427 to optionally initiate capturing link information (such as visual data, acceleration data from one or more IMU sensors, etc.) during movement of the mobile device along a path of travel away from the current acquisition location and toward the next acquisition location within the building interior. As described elsewhere herein, the captured link information may include additional sensor data (e.g., from one or more IMUs or inertial measurement units on the mobile computing device or otherwise carried by the user) and / or additional visual information recorded during such movement (e.g., panoramic images, other types of images, panoramic or non-panoramic videos, etc.), and in some embodiments, the link information may be analyzed to determine the changing pose (position and orientation) of the mobile computing device during movement, as well as information about the room shape of the enclosed room and the path of the mobile computing device during movement. Initiating the capture of such link information may be performed in response to an explicit instruction from a user of the mobile computing device or based on one or more automated analyses of information recorded from the mobile computing device. Additionally, in some embodiments, the routine may also optionally determine and provide to the user (e.g., by monitoring the movement of the mobile device) one or more guidance cues regarding the motion of the mobile device, the quality of the sensor data, and / or the visual information captured during movement to the next acquisition location, including information regarding associated lighting / environmental conditions, the desirability of capturing the next acquisition location, and any other suitable aspects of capturing link information. Similarly, the routine may optionally obtain annotations and / or other information from the user regarding the path of travel, such as for later use in presenting information regarding the path of travel or the resulting inter-panoramic image connection links. In block 429, the routine then determines that the mobile computing device (and one or more associated camera devices) has reached the next acquisition location (e.g., based on an instruction from the user, based on the user's forward movement stopping for at least a predefined amount of time, etc.) to serve as the new current acquisition location, and returns to block 415 to perform acquisition location image acquisition activities for the new current acquisition location.

[0100] If instead it is determined in box 425 that there are no more acquisition locations to acquire image information of the current building or other structure, the routine proceeds to box 430 to optionally analyze the acquisition location information of the building or other structure, such as to identify possible additional coverage (and / or other information) to be acquired within the interior of the building. For example, the ICA system may provide one or more notifications to the user regarding the information acquired during the capture of multiple acquisition locations and optionally corresponding link information, such as if it is determined that the quality of one or more segments of the recorded information is insufficient or unsatisfactory or does not appear to provide complete building coverage. After box 430, the routine continues to box 435 to optionally pre-process the acquired 360° target panoramic images, which are then subsequently used to generate related mapping information (e.g., to place the images in a spherical format, determine vanishing lines and vanishing points of the images, etc.). In box 480, these images and any associated generated or obtained information are stored for later use.

[0101] If it is instead determined in box 410 that the instructions or other information described in box 405 are not to capture images and other data representing the interior of a building, the routine instead continues to box 490 to perform any other indicated operations (such as, any housekeeping tasks) as appropriate, configure parameters for use in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device capturing the interior of one or more buildings, an operator user of an ICA system, etc.), obtain and store other information about users of the system, respond to requests for generated and stored information, etc.

[0102] After frame 480 or 490, routine advances to frame 495 to determine whether to continue, such as until receiving explicit termination instruction, or only when receiving explicit continuation instruction instead. If it is determined to continue, routine returns to frame 405 to wait for additional instructions or information, and if not, then advances to step 499 and ends.

[0103] FIG. 5A to FIG. 5B An example implementation of a flow chart for a mapping information generation manager (MIGM) system routine 500 is illustrated. The routine may be implemented by, for example Figure 1A MIGM system 160, Figure 3 388 and / or an implementation of a MIGM system as described elsewhere herein, such as to generate a floor plan and optionally other mapping information for an area based at least in part on one or more images defining the area and optionally additional data captured by a mobile computing device. FIG. 5A to FIG. 5BIn the example of , the generated mapping information includes a 2D floor plan and / or a 3D computer model floor plan of a building (such as a house), but in other embodiments, other types of mapping information may be determined and generated for other types of buildings and used in other ways, as discussed elsewhere in this document.

[0104] The illustrated implementation of the routine begins at block 505 where information or instructions are received. The routine continues to block 510 to determine whether the instructions received in block 505 indicate that mapping information for the indicated building is to be generated, and if so, the routine continues to execute blocks 515 to 588 to do so, otherwise continuing to block 590.

[0105] In block 515, the routine determines whether image information is already available for the building, or whether such information needs to be collected instead. If it is determined in block 515 that the information needs to be collected, the routine continues to block 520 to collect such information, optionally waiting for one or more users or devices to move throughout the building and collect panoramic or other images at multiple collection locations in multiple rooms of the building, and optionally further analyzing the images and / or metadata information collected thereabout to interconnect the images, as discussed elsewhere herein. Figure 4 An example implementation of an ICA system routine for performing such an image acquisition is provided. If instead it is determined at block 515 that it is not necessary to acquire an image, the routine instead continues to block 530 to obtain existing panoramic or other images from multiple acquisition locations in multiple rooms of the building, optionally along with the acquisition of interconnected information of the images and metadata information related to movement between acquisition locations, such as may have been supplied in block 505 along with corresponding instructions in some cases.

[0106] After box 520 or 530, the routine continues to box 535 to optionally obtain additional information about the building, such as from activities performed during the acquisition and optional analysis of the images, and / or from one or more external sources (e.g., an online database, information provided by one or more end users, etc.), such additional information may include, for example, the exterior dimensions and / or shape of the building, additional images and / or annotation information captured corresponding to specific locations within the building (optionally for locations different from the acquisition location of the captured panoramic or other image), additional images and / or annotation information captured corresponding to specific locations outside the building (e.g., surrounding the building and / or for other structures on the same property), etc.

[0107] After box 535, the routine continues to box 540, where it is determined whether a set of linked target panoramic images (or other images) of the building is generated, and if so, continues to box 542. In box 542, the routine selects pairs of at least some of the images (e.g., based on a pair of images with overlapping visual content), and for each pair, determines the relative direction between the images of the pair (whether directly from the acquisition position of one image to the acquisition position of the other image or via one or more other intermediate acquisition positions of the other images) based on the shared visual content and / or other captured linking information (e.g., movement information) associated with the images of the pair. In box 542, the routine further uses at least the relative direction information of the image pair to determine the global relative position of some or all of the images to each other in a common coordinate system, such as to create a virtual tour through which an end user can move from any of the images to one or more other images linked to the starting image, and similarly from the next image to one or more additional images linked to the next image. Additional details on creating such a set of linked images are included elsewhere herein.

[0108] After block 542, or if it is instead determined in block 540 that the instruction or other information received in block 505 is not to determine a set of linked images, the routine proceeds to block 545 where the routine determines whether a room shape corresponding to one or more target images of the building and its enclosed rooms is generated. If so, the routine proceeds to block 547 where the routine determines, for one or some or all rooms in the building each having one or more target images (where the capture location is within the room), a 2D and / or 3D shape of the room based at least in part on visual data of the one or more target images captured in the room by one or more camera devices and / or additional data captured in the room by a mobile computing device associated with the one or more camera devices. Determining the room shape of a room may include analyzing the visual content of one or more target images captured in the room by one or more camera devices, analyzing the visual content of one or more additional images captured in the room by an associated mobile computing device, and / or analyzing additional non-visual data captured in the room by an associated mobile computing device, including determining initial estimated acquisition position information (e.g., acquisition position and optionally acquisition orientation) for each of the target images captured in the room (e.g., for further analysis and refinement by the ILDM system). Analysis of the various data collected in the room may also include identifying features of wall structural elements of the room (e.g., windows, doorways and stairs and other inter-room wall openings and connecting passages, wall boundaries between a wall and another wall and / or ceiling and / or floor, etc.) and determining the locations of those identified features within the determined room shape of the room, optionally by generating a 3D point cloud of some or all of the room walls and optionally the ceiling and / or floor (e.g., by analyzing at least visual data of images collected in the room and optionally additional data captured by a mobile computing device, such as using one or more of SfM or SLAM or MVS analysis), and / or by determining planar surfaces corresponding to some or all of the walls and optionally the floor and / or ceiling (e.g., by determining normals / orthogonal directions to the planes of the identified features and combining such information to determine wall position hypotheses and optionally clustering multiple wall position hypotheses for a given wall to achieve a final determination of the location of that wall). Additional details on determining the room shape and identifying additional information for the room, including estimated acquisition position information for target images collected in the room, are included elsewhere herein.

[0109] After block 547, the routine continues to block 550, where the routine determines whether to further generate a floor plan of the building based at least in part on the determined room shapes from block 547 and optionally other information about how the determined room shapes are positioned relative to each other. If so, the routine continues to block 565, where the routine uses the determined room shapes to create an initial 2D floor plan, such as by connecting the inter-room corridors in their respective rooms, by optionally positioning the room shapes around the determined acquisition locations of the target image (e.g., if the acquisition locations are interconnected), and by optionally applying one or more constraints or optimizations. Such a floor plan may include, for example, relative position and shape information of various rooms without providing any actual size information of individual rooms or the building as a whole, and may further include multiple linked or associated sub-maps of the building (e.g., to reflect different floors, levels, sections, etc.). The routine further associates the locations of doors, wall openings, and other identified wall elements on the floor plan.

[0110] After box 565, the routine optionally performs one or more steps 575 to 580 to determine and associate additional information with the floor plan. In box 575, the routine optionally estimates the dimensions of some or all of the rooms, such as based on analysis of the image and / or its acquisition metadata or based on overall dimensional information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan. It will be appreciated that if sufficiently detailed dimensional information is available, architectural drawings, engineering drawings, etc. can be generated from the floor plan. After box 575, the routine continues to box 580 to optionally associate further information with the floor plan (e.g., with a specific room or other location within the building), such as additional images and / or annotation information for the specified location. In box 585, the routine further estimates the height of the walls in some or all of the rooms, such as based on analysis of the image and optionally the size of known objects in the image, and height information about the camera when the image was acquired, and further uses this information to generate a 3D computer model floor plan of the building, wherein the 2D and 3D floor plans are associated with each other.

[0111] Following box 585, or if the floor plan is not determined in box 550 or the room shape is instead determined in box 545, the routine continues to box 588 to store the generated mapping information and optionally other generated information, and optionally further use the generated mapping information, such as to provide a generated 2D floor plan and / or 3D computer model floor plan for display on one or more client devices and / or to one or more other devices for automated navigation of those devices and / or associated vehicles or other entities, to similarly provide and use information about the determined room shape and / or a set of linked panoramic images and / or about additional information determined based on the contents of the rooms and / or pathways between rooms, and the like.

[0112] If it is determined in box 510 instead that the information or instructions received in box 505 are not to generate mapping information for the indicated building, the routine instead continues to box 590 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and responding to requests for previously generated floor plans and / or other generated information (e.g., requests for such information to be used by the ILDM system, requests for such information to be displayed on one or more client devices, requests for such information to be provided to one or more other devices for automated navigation, etc.), obtaining and storing information about the building for use in later operations (e.g., information about the size, number or type of rooms, total floor area, other adjacent or nearby buildings, adjacent or nearby vegetation, external images, etc.), etc.

[0113] After frame 588 or 590, routine continues to frame 595 to determine whether to continue, such as until receiving explicit termination instruction, or only when receiving explicit continuation instruction instead. If determined to continue, routine returns to frame 505 to wait and receive additional instructions or information, otherwise, continue to frame 599 and end.

[0114] FIG. 6A to FIG. 6B An example implementation of a flow chart for an image location determination manager (ILDM) system routine 600 is illustrated. The routine may be implemented by, for example Figure 1A ILDM system 140, Figure 3 The ILDM system 340 and / or as described in relation to Figures 2A to 2U and executing execution of the ILDM system described elsewhere herein, such as performing automated operations related to determining acquisition position information (e.g., acquisition position and optionally acquisition orientation) of a target image captured by a camera device based at least in part on analysis of visual data of the target image and by using additional data captured by an associated mobile computing device, and then using the determined acquisition position information in one or more automated manners. FIG. 6A to FIG. 6B In the example of , the acquisition location information is determined relative to a floor plan of a building (such as a house), but in other embodiments, other types of mapping information may be used for other types of structures or for non-structural locations, and the determined acquisition location information may be used in other ways than discussed with respect to routine 600, as discussed elsewhere herein.

[0115] The illustrated implementation of the routine begins at block 605 where information or instructions are received. The routine continues to block 610 to determine whether the instructions received in block 605 indicate determining acquisition location information for a target image captured for an indicated building, and if so, the routine continues to execute blocks 615 to 688 to do so, otherwise continuing to block 690.

[0116] In block 615, the routine obtains a target image to be analyzed (e.g., the 360° target panoramic image provided with the information of block 605), and determines a first estimated room shape of a room enclosing the capture location of the target image based on the visual data of the target image. In at least some embodiments, the determination of the estimated room shape information from the target image may include calling the MIGM routine and providing the target image as input along with instructions to determine its room shape, and receiving the first estimated room shape as output of the routine (wherein FIG. 5A to FIG. 5B An example of such a routine is provided), and in some embodiments may include using one or more trained neural networks to generate a first estimated room shape from visual data of a target image.

[0117] After block 615, the routine continues to block 620 where the routine determines whether to obtain a second estimated room shape for the enclosed room from an existing partial or full floor plan of the building or instead by analyzing additional data captured in the enclosed room by an associated mobile computing device. If it is determined in box 620 to use an existing floor plan, the routine continues to box 675, where the routine retrieves a partial or full floor plan of the building that includes information about 2D and / or 3D room shapes of some or all of the rooms of the building (and optionally includes additional information for a particular room, such as acquisition location information of other existing images acquired in the room, locations of structural wall elements in the room, and optionally other information acquired in the room, etc.), and selects one or more of those room shapes as one or more candidates for enclosing the room (e.g., selecting a room shape for a particular room identified by a user operating a mobile computing device and / or camera device, selecting a room shape for a particular room identified by other data obtained from the mobile computing device and / or camera device, selecting one or more rooms as possible candidates based on other information obtained during acquisition of the target image and / or additional data, etc.). After block 675, the routine continues to block 680 where the routine uses the first estimated room shape information from block 615 and additional information for one or more candidate room shapes from block 675 to determine an offset of a matching feature (e.g., a structural wall element) identified in the first estimated room shape and one of the candidate room shapes, and uses the determined offset information to determine where the acquisition position and, optionally, the acquisition orientation of the camera device is located within the candidate room shape. If multiple candidate room shapes are available, an analysis may be performed for each of those multiple candidate room shapes where the best match (e.g., having the smallest aggregate offset) is identified between the first estimated room shape and one of the candidate room shapes, and the determined acquisition position and, optionally, the acquisition orientation of the identified candidate room shape is selected for future use. The candidate room shape from the floor plan used to determine the acquisition position of the target image is further selected for future use as the final room shape of the enclosed room.

[0118] Alternatively, if it is determined in block 620 that an existing floor plan is not to be used (e.g., if such a floor plan has not yet been generated), the routine instead continues to block 625, where the routine retrieves additional data captured by the mobile computing device in the enclosed room (e.g., one or more other images and optionally associated acquisition metadata, such as IMU data, SLAM-based tracking data, sensed depth data, etc.), such as additional data captured by a camera device while capturing a target image and while the mobile computing device is located near or at the capture location of the target image, or instead at one or more other times and / or at one or more other locations within the enclosed room. In block 625, the routine also optionally uses the additional data from the mobile computing device to determine a second estimated room shape for the enclosed room. In at least some embodiments, determination of room shape information from the additional data captured by the mobile computing device may include calling a MIGM routine and providing the additional data as input along with instructions to determine its room shape and receiving a second estimated room shape as an output of the routine (wherein FIG. 5A to FIG. 5B An example of such a routine is provided), and further determining one or more positions and optionally orientations of the mobile computing device for one or more other images captured in the enclosed room.

[0119] After box 625, the routine continues to box 635, where the routine determines one or more estimates of the acquisition position and optionally the acquisition orientation of the target image by performing one or more of operations 641 to 649, such as for one or more other images, relative to the determined position and optionally the orientation of the mobile computing device, and in some embodiments and situations, includes using the input of one or more of operations 641 to 649 as an initial estimate, which is provided as input to one or more other operations 641 to 649, which provide further updates to those initial estimates, as discussed in more detail elsewhere in this document. In addition, some or all of operations 641 to 649 may also include generating confidence level information in the estimated acquisition position information generated by the operation, such as for use later in box 660. In particular, in some embodiments and scenarios, the routine may perform the operations of box 641 (whether in addition to or in place of other operations 643 to 649) to analyze visual data of one or more other images to determine the location of the camera device in the visual data of those other images, such as by identifying one or more markers located on the camera device in the visual data and / or by performing object detection and recognition of the shape of the camera device in the visual data, and use the determined location information in the visual data to determine the acquisition location information of the camera device relative to the mobile computing device from the one or more other images. In addition, in some embodiments and scenarios, the routine may perform the operations of box 643 (whether in addition to or in place of other operations 641 to 649) to analyze movement data of the mobile computing device to and / or from the location where the camera device acquires the target image, and perform a determination of the camera device acquisition location relative to the mobile computing device from the movement data, such as using pattern analysis of the movement data (e.g., performed by one or more trained neural networks). Additionally, in some embodiments and scenarios, the routine may perform the operations of block 645 (whether in addition to or in place of the other operations 641 to 649) to use the first and second estimated room shape information from blocks 615 and 625 to determine offsets of matching features (e.g., structural wall elements) identified in those room shapes, and use the offset information to determine an acquisition position and, optionally, an acquisition orientation of the camera device relative to the mobile computing device, such as by determining that the acquisition position information associated with the first estimated room shape is located within the second estimated room shape. Additionally, in some embodiments and scenarios, the routine may perform the operations of block 647 (whether in addition to or in place of the other operations 641 to 649) to analyze the visual data of the target image and one or more other images to identify matching features visible in the visual data of the target image and the other images, determine offsets of those identified features in the visual data of the different images, and use the determined offset information to determine an acquisition position and, optionally, an acquisition orientation of the camera device relative to the mobile computing device.Additionally, in some embodiments and scenarios, the routine may perform the operations of box 649 (whether in addition to or in place of other operations 641 to 647) to analyze the visual data of the target image and one or more other previous images captured by the camera device in the same room (e.g., one or more other previously captured target images for which capture position information has been previously determined) to identify matching features visible in the data of the target image and the other previous images, determine the offsets of those identified features in the visual data of the different images, and use the determined offset information to determine the capture position and, optionally, the capture orientation of the camera device relative to the other previous images. Additionally, the routine may further optionally determine a common coordinate system for the visual data of the target image and one or more other previous images, such as to enable the use of the combined visual data to generate or improve an estimated room shape for an enclosed room.

[0120] After performing one or more of operations 641 to 649 (e.g., in some embodiments and scenarios, a single one of the operations, and in other embodiments and scenarios, some or all of the operations), the routine continues to box 660, where the routine selects and uses the estimated acquisition position information of the target image from a single one of operations 641 to 649 (e.g., if only one operation is performed, or if multiple operations are performed and a single operation is selected for use, such as based on having the highest associated confidence level information or otherwise satisfying one or more defined criteria), or instead combines the estimated acquisition position information of the target image from multiple ones of the performed operations 641 to 649 (e.g., for the performed operations 641 to 649, some or all of those performed operations). Combining the estimated information from multiple operations can include, for example, using the associated confidence level information of the multiple operations as part of the combination, such as to perform a weighted average, as part of a statistical analysis (e.g., to identify and exclude one or more outliers), etc. After block 660 , the routine continues to block 665 where the routine determines a final room shape for enclosing the room further based on the first and / or second estimated room shapes from blocks 615 and 625 .

[0121] After either block 665 or 680, the routine continues to block 685 where the final determined acquisition location information (acquisition location and optionally acquisition orientation) of the target image from block 680 or block 660 is added to the final room shape (e.g., added to an existing floor plan, if one exists, associated with the final room shape for inclusion in a partial or full floor plan of a building, if generated based at least in part on the target image, etc.). After block 685, the routine continues to block 688 where the routine stores the information determined and generated in blocks 615 through 685 and optionally displays some or all of the determined acquisition location information (e.g., acquisition location) of the target image in its enclosed room (e.g., in the final room shape of the enclosed room, such as on a mobile computing device).

[0122] If it is instead determined in block 610 that the information or instructions received in block 605 are not acquisition location information for determining a target image, the routine instead continues to block 690 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and responding to a request for a previously determined image acquisition location and / or for an associated target image (e.g., a request for such information to be displayed on one or more client devices, a request for such information to be provided to one or more other devices for automated navigation, etc.), obtaining and storing information about a building for later operation (e.g., information about a floor plan and associated wall element locations of rooms in the floor plan, etc.), etc.

[0123] After frame 688 or 690, routine continues to frame 695 to determine whether to continue, such as until receiving explicit termination instruction, or only when receiving explicit continuation instruction instead. If determined to continue, routine returns to frame 605 to wait and receive additional instructions or information, otherwise, continues to frame 699 and ends.

[0124] Figure 7 An example implementation of a flow chart for a building map viewer system routine 700 is illustrated. The routine may be executed, for example, Figure 1A The map viewer client computing device 175 and its software system (not shown), Figure 3 The client computing device 390 and / or the mobile computing device 360, and / or a mapping information viewer or presentation system as described elsewhere herein, may be used to perform, such as: receiving and displaying mapping information (e.g., a 2D or 3D floor plan, etc.) of a defined area, the mapping information including a visual indication of one or more determined image acquisition locations; and optionally displaying additional information (e.g., an image) associated with a specific location in the mapping information. Figure 7In the example shown, the mapping information presented is for the interior of a building (such as a house), but in other embodiments, other types of mapping information may be presented and used in other ways for other types of buildings or environments, as discussed elsewhere in this document.

[0125] The illustrated implementation of the routine begins at block 705, where an instruction or information is received. At block 710, the routine determines whether the received instruction or information indicates to display or otherwise present information representing the interior of the building, and if not, continues to block 790. Otherwise, the routine proceeds to block 712 to retrieve an indication of a floor plan and / or other generated mapping information of the building and optionally associated linked information of surrounding locations within the building and / or outside the building, and selects an initial view of the retrieved information (e.g., a view of the floor plan). In block 715, the routine then displays or otherwise presents a current view of the retrieved information and waits for a user selection in block 717. After the user selection in block 717, if it is determined in block 720 that the user selection corresponds to the current building area (e.g., to change the current view), the routine continues to block 722 to update the current view according to the user selection, and then returns to block 715 to update the displayed or otherwise presented information accordingly. User selections and corresponding updates to the current view may include, for example, displaying or otherwise presenting a piece of associated linked information selected by the user (e.g., a particular image associated with a displayed visual indication of a determined acquisition location), changing the manner in which the current view is displayed (e.g., zooming in or out; rotating information where appropriate; selecting a new portion of the floor plan to be displayed or otherwise presented, such as where some or all of the new portion was not previously visible, or instead where the new portion is a subset of previously visible information; etc.).

[0126] If it is instead determined in box 710 that the instructions or other information received in box 705 will not present information representing the interior of a building, the routine instead continues to box 790 to perform any other indicated operations (such as, any housekeeping tasks) as appropriate, configure parameters for use in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device capturing the interior of one or more buildings, an operator-user of an ILDM system, etc.), obtain and store other information about users of the system, respond to requests for generated and stored information, etc.

[0127] After box 790, or if it is determined in box 720 that the user selection does not correspond to the current building area, the routine proceeds to box 795 to determine whether to continue, such as until an explicit termination instruction is received, or instead only if an explicit continuation instruction is received. If it is determined to continue (including if the user makes a selection related to a new location to be presented in box 717), the routine returns to box 705 to wait for additional instructions or information (or if the user makes a selection related to a new location to be presented in box 717, then directly proceed to box 712), and if not, proceed to step 799 and end.

[0128] Non-exclusive example implementations described herein are further described in the following clauses.

[0129] A01. A computer-implemented method comprising:

[0130] obtaining, by a mobile computing device having one or more image sensors and one or more inertial measurement unit (IMU) sensors and in a first area of ​​a first room of a building and being one of the one or more computing devices, a first panoramic image captured in the first area by a camera device separate from the mobile computing device and having 360 degrees horizontal coverage about a vertical axis, and first additional data captured in the first area by the mobile computing device and including motion data from the IMU sensor and additional visual data from the image sensor;

[0131] obtaining, by the mobile computing device, movement data captured by the mobile computing device as it moves along a travel path from the first area of ​​the first room to a second area of ​​a second room of the building;

[0132] obtaining, by the mobile computing device, a second panoramic image captured by the camera device in the second area of ​​the second room and having 360-degree horizontal coverage about a vertical axis, and second additional data captured by the mobile computing device in the second area and including motion data from the IMU sensor and other visual data from the image sensor;

[0133] determining, by the mobile computing device, a first position and orientation of the mobile computing device in the first room and a second position and orientation of the camera device in the first room based at least in part on analyzing the motion data and the additional visual data of the first additional data, the second position and orientation being relative to the first position and orientation of the mobile computing device and based at least in part on the visual data of the first panoramic image;

[0134] determining, by the mobile computing device, a third position and orientation of the mobile computing device in the second room and a fourth position and orientation of the camera device in the second room based at least in part on analyzing the motion data and the other visual data of the second additional data, the fourth position and orientation being relative to the third position and orientation of the mobile computing device and based at least in part on the visual data of the second panoramic image;

[0135] determining, by the mobile computing device, a position of the first panoramic image within a three-dimensional (3D) first shape of the first room determined at least in part based on the visual data of the first panoramic image using the second position and orientation of the camera device, and determining a position of the second panoramic image within a 3D second shape of the second room determined at least in part based on the visual data of the second panoramic image using the fourth position and orientation of the camera device;

[0136] generating, by the mobile computing device and using the 3D first shape of the first room and the 3D second shape of the second room and the determined positions of the first panoramic image and the second panoramic image, a local floor plan of the building, including determining the relative positions of the 3D first shape of the first room and the 3D second shape of the second room using the movement data of the mobile computing device; and

[0137] The generated local floor plan of the building with a visual indication overlay is displayed by the mobile computing device and to a user located in the building to show the determined locations of the first panoramic image and the second panoramic image in the respective first room and the second room to enable determination of additional areas of the building for which additional panoramic images are to be acquired.

[0138] A02. A computer-implemented method comprising:

[0139] Obtained by a computing device that is one of the one or more computing devices

[0140] a first panoramic image captured by a camera device in a first area of ​​a first room of a building and having visual coverage of surrounding walls of the first room, and first additional data captured in the first area by a mobile computing device separate from the camera device;

[0141] obtaining, by the computing device, movement data captured by the mobile computing device when the mobile computing device moves from the first area of ​​the first room to a second area of ​​a second room of the building;

[0142] obtaining, by the computing device, a second panoramic image captured by the camera device in the second area of ​​the second room and having visual coverage of surrounding walls of the second room, and second additional data captured by the mobile computing device in the second area;

[0143] determining, by the computing device and using a first position of the mobile computing device in the first room based at least in part on the first additional data, a second position of the camera device in the first room relative to the first position of the mobile computing device, and determining, using the second position, a position of the first panoramic image within a first shape of the first room, the first shape being based at least in part on the visual coverage of the first panoramic image;

[0144] determining, by the computing device and using a third position of the mobile computing device in the second room based at least in part on the second additional data, a fourth position of the camera device in the second room relative to the third position of the mobile computing device, and determining a position of the second panoramic image within a second shape of the second room using the fourth position, the second shape being based at least in part on the visual coverage of the second panoramic image;

[0145] generating, by the computing device and using the first shape of the first room and the second shape of the second room and the determined positions of the first panoramic image and the second panoramic image, at least a partial floor plan of the building, including using the movement data of the mobile computing device to determine the relative positions of the first shape of the first room and the second shape of the second room; and

[0146] The generated at least partial floor plan of the building with a visual indication overlay is displayed by the computing device to illustrate the determined positions of the first panoramic image and the second panoramic image in the respective first room and the second room.

[0147] A03. A computer-implemented method for one or more computing devices to perform automated operations, the automated operations comprising at least:

[0148] obtaining, by the one or more computing devices, one or more panoramic images having visual data captured by a camera device in one or more rooms of a building and additional visual data captured in the one or more rooms by a mobile computing device separate from the camera device;

[0149] determining, by the one or more computing devices and for each of the one or more rooms, a shape of the room using at least one of the visual data of the one or more panoramic images or the additional visual data captured by the mobile computing device;

[0150] determining, by the one or more computing devices and for each of the one or more rooms, a first position of the mobile computing device in the room to capture at least some of the additional visual data from the room, and determining a second position of the camera device in the room to capture at least one of the panoramic images, wherein the second position of the camera device in the room is determined relative to the first position of the mobile computing device in the room;

[0151] generating, by the one or more computing devices, mapping information of the one or more rooms, the mapping information including, for each of the one or more rooms, a position of the at least one panoramic image captured in the room within the determined shape of the room, including using the second position of the camera device in the room as part of determining the position of the at least one panoramic image captured in the room; and

[0152] The generated mapping information of the one or more rooms is provided by the one or more computing devices to enable display of the determined shape of each of the one or more rooms with one or more superimposed visual indications to show the location of the at least one panoramic image in each of the one or more rooms.

[0153] A04. A computer-implemented method for one or more computing devices to perform automated operations, the automated operations comprising at least:

[0154] obtaining a plurality of images having visual data captured by a camera device in two or more rooms of a building and additional visual data captured in the two or more rooms by a mobile computing device separate from the camera device;

[0155] determining a shape of each of the two or more rooms using at least one of the visual data of the plurality of images or the additional visual data captured by the mobile computing device;

[0156] determining, for each of the two or more rooms, a first position of the mobile computing device in the room during capturing some of the additional visual data from the room and based at least in part on the some additional visual data, and a second position of the camera device in the room during capturing at least one of the plurality of images in the room, wherein the second position in the room is determined relative to the first position in the room;

[0157] generating mapping information including the determined shape of each of the two or more rooms positioned relative to each other and, for each of the two or more rooms, a position of the at least one image captured in the room within the determined shape of the room, including using the second position of the camera device in the room as part of determining the position of the at least one image of the room; and

[0158] The generated mapping information of the one or more rooms is provided to enable display of the generated mapping information.

[0159] A05. The computer-implemented method of any of clauses A01 to A04, further comprising, after displaying the generated partial floor plan of the building to the user located in the building:

[0160] receiving, by the mobile computing device from the user, an indication of an additional third area in a third room of the building;

[0161] obtaining, by the mobile computing device, a third panoramic image captured by the camera device in the third area, and third additional data captured by the mobile computing device in the third area and including motion data from the IMU sensor and visual data from the image sensor;

[0162] determining, by the mobile computing device, a position of the third panoramic image within a third shape of the third room, the third shape being based on a combination of the visual data of the third panoramic image and the third additional data;

[0163] updating, by the mobile computing device, the partial floor plan of the building to determine a position of the third shape of the third room relative to at least one of the 3D first shape of the first room and the 3D second shape of the second room; and

[0164] The updated partial floor plan of the building is displayed by the mobile computing device.

[0165] A06. The computer-implemented method of clause A05, further comprising, before generating the partial floor plan:

[0166] determining, by the mobile computing device, the 3D first shape of the first room based on a combination of the visual data of the first panoramic image captured by the camera device and the first additional data captured by the mobile computing device in the first area;

[0167] determining, by the mobile computing device, the 3D second shape of the second room based on a combination of the visual data of the first panoramic image captured by the camera device and the first additional data captured by the mobile computing device in the first area; and

[0168] The third shape of the third room is determined by the mobile computing device based on a combination of the visual data of the third panoramic image captured by the camera device and the third additional data captured by the mobile computing device in the third area, wherein the third shape is a 3D room shape.

[0169] A07. A computer-implemented method as described in any of clauses A01 to A06, further comprising determining, by the mobile computing device and at least in part based on analyzing the visual data of the first panoramic image captured in the first room and the visual data of the second panoramic image captured in the second room, the location of a doorway between the first room and the second room, and wherein generating the local floor plan is also based in part on the determined location of the doorway between the first room and the second room.

[0170] A08. A computer-implemented method as described in any one of clauses A01 to A07,

[0171] wherein the computing device is the mobile computing device, wherein displaying the generated at least partial floor plan comprises displaying the generated at least partial floor plan to a user of the mobile computing device in the building, and wherein the method further comprises, after displaying the generated at least partial floor plan:

[0172] receiving, by the mobile computing device from the user, an indication of a third room of the building separate from the first room and the second room;

[0173] obtaining, by the mobile computing device, a third panoramic image captured by the camera device in the third room, and third additional data captured by the mobile computing device in the third room and comprising motion data from one or more inertial measurement unit (IMU) sensors of the mobile computing device and visual data from one or more image sensors of the mobile computing device;

[0174] determining, by the mobile computing device, a position of the third panoramic image within a third shape of the third room, the third shape being determined based on a combination of visual data of the third panoramic image and the third additional data captured in the third room;

[0175] updating, by the mobile computing device, the at least partial floor plan to include a position of the third shape of the third room relative to at least one of the first shape of the first room and the second shape of the second room; and

[0176] The updated at least partial floor plan of the building is displayed by the mobile computing device.

[0177] A09. A computer-implemented method as described in clause A08,

[0178] Also included, before generating the at least partial floor plan:

[0179] determining, by the mobile computing device, the first shape of the first room based on a combination of visual data of the first panoramic image captured by the camera device and based on the first additional data captured by the mobile computing device, wherein the first shape is a first three-dimensional (3D) shape;

[0180] determining, by the mobile computing device, the second shape of the second room based on a combination of visual data of the second panoramic image captured by the camera device and based on the second additional data captured by the mobile computing device, wherein the second shape is a second 3D shape; and

[0181] determining, by the mobile computing device, the third shape of the third room based on a combination of the visual data of the third panoramic image captured by the camera device and based on the third additional data captured by the mobile computing device, wherein the third shape is a third 3D shape,

[0182] And wherein generating the at least partial floor plan of the building includes positioning, by the mobile computing device, the first 3D shape and the second 3D shape relative to each other in the generated at least partial floor plan.

[0183] A10. A computer-implemented method as described in any of clauses A01 to A09,

[0184] wherein the first panoramic image captured by the camera device in the first area includes visual data of some or all of the first room, wherein the first additional data captured by the mobile computing device in the first area includes additional visual data of at least some of the first room, and wherein the method further comprises using at least one of the following operations (a) to (g):

[0185] (a) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the additional visual data to identify the camera device in the additional visual data, and using the position of the identified camera device in the additional visual data as part of determining the second position of the camera device in the first room; or

[0186] (b) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the additional visual data to identify one or more markers in the additional visual data located on the camera device, and using one or more positions of the identified one or more markers in the additional visual data as part of determining the second position of the camera device in the first room; or

[0187] (c) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the visual data of the first panoramic image to generate a first estimated shape of the room, analyzing the additional visual data to generate a second estimated shape of the room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape at which the visual data of the first panoramic image was captured; or

[0188] (d) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the visual data of the first panoramic image to generate a first estimated shape of the room, analyzing other depth data captured by the mobile computing device in the first room to generate a second estimated shape of the room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape at which the visual data of the first panoramic image was captured; or

[0189] (e) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the visual data and the additional visual data of the first panoramic image to identify features visible in both the visual data and the additional visual data, and using the positions of the identified features as part of determining the second position of the camera device in the first room; or

[0190] (f) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing movement data of the mobile computing device in the room as the mobile computing device moves to and / or from the position at which the camera device captured the first panoramic image to identify the position, and using the identified position as part of determining the second position of the camera device in the first room; or

[0191] (g) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, including analyzing visual data of the first panoramic image and other visual data of another panoramic image captured in the first room to identify features visible in both the first panoramic image and the other panoramic image, and using the positions of the identified features to determine a common coordinate system for the first panoramic image and the other panoramic image for combining the visual data and the other visual data and as part of determining the second position of the camera device in the first room.

[0192] A11. A computer-implemented method as described in clause A10, wherein using at least one of the operations (a) to (g) includes: the computing device combining information from multiple operations (a) to (g) by using confidence information generated for the multiple analysis operations or at least one of the statistical analyses performed on the multiple analysis operations.

[0193] A12. A computer-implemented method as described in any of clauses A01 to A11,

[0194] Also includes at least one of the following:

[0195] generating the at least partial floor plan based at least in part on motion data obtained in the first room using one or more inertial measurement unit (IMU) sensors of the mobile computing device and visual data obtained in the first room using one or more image sensors of the mobile computing device, including analyzing the motion data and the visual data to estimate a three-dimensional (3D) shape of the first room, analyzing additional visual data of the first panoramic image to identify one or more structural wall elements of a wall of the first room, generating one or more locations of the identified one or more structural wall elements of the wall of the first room within the estimated 3D shape of the first room, and using the generated one or more locations as part of generating the at least partial floor plan; or

[0196] generating, performed by the computing device, the at least partial floor plan based at least in part on a plurality of panoramic images obtained at a plurality of acquisition locations, including generating positions of the plurality of panoramic images relative to each other by determining directions between pairs of the plurality of panoramic images based on at least one of analyzing visual data of the plurality of panoramic images or analyzing movement data from the one or more IMU sensors as the mobile computing device moves between acquisition locations of the plurality of panoramic images; or

[0197] Performing generation of the at least partial floor plan based at least in part on a combination of operations by the mobile computing device and at least one additional computing device separate from the mobile computing device includes generating, by the mobile computing device, an initial estimated shape of the first room based on at least one of the additional visual data of the first panoramic image or the visual data captured by the mobile computing device in the first room, and including using, by the at least one additional computing device, the first shape of the first room to improve the initial estimated shape of the first room based at least in part on performing further analysis of the additional data.

[0198] A13. A computer-implemented method as described in any of clauses A01 to A12, wherein obtaining the one or more panoramic images and the additional visual data includes obtaining a plurality of panoramic images, the plurality of panoramic images including at least one panoramic image in each of two or more rooms of the building, and includes obtaining, by the mobile computing device, additional visual data in each of the two or more rooms, and wherein generating the mapping information includes generating, by the one or more computing devices, at least a partial floor plan of the building, the at least partial floor plan including determined shapes of the two or more rooms positioned relative to each other.

[0199] A14. The computer-implemented method of clause A13, wherein the mobile computing device is the one or more computing devices that perform the automated operation, and wherein the method further comprises:

[0200] capturing, by the mobile computing device, movement data during travel between the two or more rooms; and

[0201] performing, by the mobile computing device, positioning the determined shapes of the two or more rooms relative to each other based at least in part on the captured movement data,

[0202] And wherein providing the generated mapping information includes displaying, by the mobile computing device, the generated at least partial floor plan of the building to a user to enable identification of additional areas of the building for acquisition of additional panoramic images.

[0203] A15. The computer-implemented method of clause A14, wherein providing the generated mapping information comprises presenting, by the one or more computing devices, the generated at least partial floor plan to a user, and wherein the automated operation further comprises, after presenting:

[0204] receiving, by the one or more computing devices from the user, an indication of an additional room of the building separate from the two or more rooms;

[0205] obtaining, by the one or more mobile computing devices, panoramic images additionally captured by the camera device in the additional room and additional data captured by the mobile computing devices in the additional room and including motion data from one or more inertial measurement unit (IMU) sensors of the mobile computing devices and visual data from one or more image sensors of the mobile computing devices;

[0206] determining, by the one or more mobile computing devices, a position of the additional panoramic image within a shape of the additional room, the shape of the additional room being determined based on a combination of visual data of the additional panoramic image and the additional data captured in the additional room;

[0207] updating, by the one or more computing devices, the at least partial floor plan to position the shape of the additional room relative to at least one of the positioned determined shapes of the two or more rooms; and

[0208] The updated at least partial floor plan of the building is presented by the one or more computing devices.

[0209] A16. The computer-implemented method of any of clauses A13 to A15, wherein the automated operation further comprises, prior to generating the at least partial floor plan:

[0210] generating, by the one or more computing devices, a first three-dimensional (3D) shape of a first room of the two or more rooms based on a combination of visual data of the at least one panoramic image captured in the first room and based on the additional visual data captured in the first room by the mobile computing device, including determining at least one of a 3D point cloud modeling at least some walls of the first room or a planar surface representing the at least some walls of the first room, and also including identifying a doorway between the first room and the second room; and

[0211] generating, by the one or more computing devices, a second 3D shape of a second room of the two or more rooms based on a combination of visual data of the at least one panoramic image captured in the second room and based on the additional visual data captured in the second room by the mobile computing device; and

[0212] wherein generating the at least partial floor plan of the building includes positioning, by the one or more computing devices, the first 3D shape and the second 3D shape relative to each other in the generated at least partial floor plan based at least in part on the location of the determined doorway between the first room and the second room.

[0213] A17. A computer-implemented method as described in any of clauses A01 to A16, wherein the mobile computing device is one or more computing devices, and wherein determining the shape of each of the one or more rooms also includes determining, by the mobile computing device, a three-dimensional (3D) shape of the room based on a combination of the visual data of the at least one panoramic image captured in the room and some of the additional visual data captured by the mobile computing device in the room.

[0214] A18. A computer-implemented method as described in clause A17, wherein obtaining the additional visual data also includes obtaining, by the mobile computing device, additional motion data captured by one or more inertial measurement units (IMUs) of the mobile computing device in the two or more rooms, and wherein determining the shape of each of the one or more rooms is further based in part on the additional motion data.

[0215] A19. A computer-implemented method as described in any of clauses A17 to A18, wherein the one or more panoramic images each include 360 ​​degrees of horizontal visual coverage around a vertical axis, and wherein generating the first 3D shape of the first room is performed without using any depth information from any depth sensing sensor for measuring distances from the position of the camera device or the mobile computing device in the first room to surrounding surfaces.

[0216] A20. The computer-implemented method of any of clauses A01 to A19, further comprising at least one of the following operations (a) to (g):

[0217] (a) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the additional visual data captured by the mobile computing device in the room to identify the camera device in the additional visual data, and using the position of the identified camera device in the additional visual data as part of determining the second position of the camera device in the room; or

[0218] (b) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the additional visual data captured by the mobile computing device in the room to identify one or more markers in the additional visual data located on the camera device, and using one or more positions of the identified one or more markers in the additional visual data as part of determining the second position of the camera device in the room; or

[0219] (c) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the visual data of the at least one panoramic image captured by the camera device in the room to generate a first estimated shape of the room, analyzing the additional visual data captured by the mobile computing device in the room to generate a second estimated shape of the room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape of the visual data capturing the at least one panoramic image in the room; or

[0220] (d) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the visual data of the at least one panoramic image captured by the camera device in the room to generate a first estimated shape of the room, analyzing other depth data captured by the mobile computing device in the room to generate a second estimated shape of the room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape of the visual data capturing the at least one panoramic image in the room; or

[0221] (e) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the visual data of the at least one panoramic image captured by the camera device in the room and the additional visual data captured by the mobile computing device in the room to identify features visible in both the visual data and the additional visual data, and using the positions of the identified features as part of determining the second position of the camera device in the room; or

[0222] (f) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing movement data of the mobile computing device in the room as the mobile computing device moves to and / or from the position where the camera device captured the at least one panoramic image in the room to identify the position, and using the identified position as part of determining the second position of the camera device in the room; or

[0223] (g) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing visual data of two or more panoramic images captured in the room to identify features visible in both of the two or more panoramic images, and using the positions of the identified features to determine a common coordinate system for the two or more panoramic images and as part of determining the second position of the camera device in the room.

[0224] A21. A computer-implemented method as described in clause A20, wherein the mobile computing device is the one or more mobile computing devices, and wherein, for each of the one or more rooms, using at least one of the operations (a) to (g) includes: the one or more computing devices combining information from multiple operations of the operations (a) to (g) for the room by using at least one of confidence information generated for multiple analysis operations or statistical analyses performed for the multiple analysis operations.

[0225] A22. A computer-implemented method as described in any of clauses A20 to A21, wherein, for each of the one or more rooms, determining the second position of the camera device in the room relative to the first position of the mobile computing device in the room also includes determining the orientation of the camera device in each of the one or more rooms during capture of each of the at least one panoramic image in the room, wherein generating the mapping information is further based in part on the determined orientation for each of the one or more rooms and each of the at least one panoramic image captured in the room, and wherein using at least one of the operations (a) to (g) for each of the one or more rooms includes combining, by the one or more computing devices, information from all of the operations (a) to (g) for the room.

[0226] A23. A computer-implemented method as described in any of clauses A01 to A22, wherein the automated operation further comprises at least one of the following:

[0227] generating the mapping information based at least in part on motion data obtained using one or more inertial measurement unit (IMU) sensors of the mobile computing device in the one or more rooms, including analyzing the additional visual data and the motion data captured by the mobile computing device in one of the one or more rooms to estimate a three-dimensional (3D) shape of the one room, analyzing the visual data of the at least one panoramic image captured in the one room to identify one or more structural wall elements of a wall of the one room, generating locations of the identified one or more structural wall elements of the wall of the one room within the estimated 3D shape of the one room, and using the generated locations as part of generating the mapping information; or

[0228] generating, performed by the one or more computing devices, the mapping information based at least in part on a plurality of panoramic images obtained at a plurality of acquisition locations, including generating positions of the plurality of panoramic images relative to each other by determining directions between pairs of the plurality of panoramic images based on at least one of analyzing visual data of the plurality of panoramic images or analyzing movement data from the one or more IMU sensors as the mobile computing device moves between acquisition locations of the plurality of panoramic images; or

[0229] Performing generation of the mapping information based at least in part on a combination of operations by the mobile computing device and at least one of the one or more computing devices separate from the mobile computing device includes generating, by the mobile computing device, an initial estimated shape of one of the multiple rooms based at least in part on the visual data of the at least one panoramic image captured in the one room or at least one of the additional visual data captured by the mobile computing device in the one room, and using, by the at least one computing device separate from the mobile computing device, the determined shape of the one room to improve the initial estimated shape of the one room based at least in part on performing further analysis on the additional data.

[0230] A24. The computer-implemented method of any of clauses A01 to A23, wherein generating the mapping information comprises generating, by the one or more computing devices, at least a partial floor plan of the building, wherein the mobile computing device is the one or more computing devices that perform the automated operation, and wherein the method further comprises, by the mobile computing device:

[0231] capturing movement data as the mobile computing device moves between the two or more rooms; and

[0232] performing positioning of the determined shapes of the two or more rooms relative to each other based at least in part on the captured movement data,

[0233] And wherein providing the generated mapping information includes displaying, by the mobile computing device, the generated at least partial floor plan of the building to a user to enable identification of additional areas of the building for acquisition of additional images.

[0234] A25. A computer-implemented method as described in any of clauses A01 to A24, wherein the multiple images are multiple panoramic images, wherein obtaining the additional visual data also includes obtaining, by the one or more computing devices, additional motion data captured by one or more inertial measurement units (IMUs) of the mobile computing device in the two or more rooms, wherein determining the shapes of the two or more rooms is based on the visual data of the multiple panoramic images and a combination of the additional motion data and the additional visual data obtained by the mobile computing device, wherein determining the first position of the mobile computing device in each of the two or more rooms also includes determining a first orientation of the mobile computing device during capture of the some additional visual data from the room, wherein determining the second position of the camera device in each of the two or more rooms also includes determining a second orientation of the camera device during capture of the at least one image captured in the room, and wherein generating the mapping information is further based in part on the determined first orientation and second orientation of each of the two or more rooms.

[0235] A26. A computer-implemented method comprising a plurality of steps to perform automated operations implementing the described techniques substantially as disclosed herein.

[0236] B01. A non-transitory computer-readable medium storing executable software instructions and / or other stored content, wherein the executable software instructions and / or other stored content cause one or more computing systems to perform automated operations, wherein the automated operations implement the method described in any one of clauses A01 to A26.

[0237] B02. A non-transitory computer-readable medium having stored thereon executable software instructions and / or other stored content that cause one or more computing systems to perform automated operations that implement substantially the described techniques disclosed herein.

[0238] C01. One or more computing systems, comprising one or more hardware processors and one or more memories storing instructions, wherein the instructions, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations, wherein the automated operations implement the method described in any one of clauses A01 to A26.

[0239] C02. One or more computing systems comprising one or more hardware processors and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations that implement the described techniques substantially as disclosed herein.

[0240] D01. A computer program, which is suitable for performing the method described in any one of clauses A01 to A26 when the computer program is run on a computer.

[0241] Aspects of the present disclosure are described herein with reference to the flowchart illustrations and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It will be appreciated that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions. It will be further appreciated that in some embodiments, the functions provided by the routines discussed above can be provided in an alternative manner, such as being divided into more routines or merged into fewer routines. Similarly, in some embodiments, the routines described can provide more or less functions than the functions described, such as when other routines described are changed to lack or include such functions accordingly, or when the amount of the functions provided is changed. In addition, although various operations can be described as being performed in a particular manner (e.g., serial or parallel or synchronous or asynchronous) and / or in a particular order, in other embodiments, operations can be performed in other orders and in other ways. Any data structure discussed above can also be structured in different ways, such as by dividing a single data structure into multiple data structures and / or by merging multiple data structures into a single data structure. Similarly, in some embodiments, described data structures may store more or less information than described, such as when other described data structures are correspondingly modified to lack or include such information, or when the amount or type of information stored is altered.

[0242] In light of the foregoing, it will be appreciated that, although specific embodiments have been described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited except by the corresponding claims and the elements recited by those claims. Additionally, although certain aspects of the present invention may be presented in certain claim forms at certain times, the inventors contemplate various aspects of the present invention in any available claim form. For example, although only some aspects of the present invention may be recited as embodied in a computer-readable storage medium at a particular time, other aspects may also be so embodied.

Claims

1. A computer-implemented method comprising: obtaining, by a computing device, a first panoramic image captured by a camera device in a first area of ​​a first room of a building and having visual coverage of surrounding walls of the first room, and first additional data captured in the first area by a mobile computing device separate from the camera device; obtaining, by the computing device, movement data captured by the mobile computing device when the mobile computing device moves from the first area of ​​the first room to a second area of ​​a second room of the building; obtaining, by the computing device, a second panoramic image captured by the camera device in the second area of ​​the second room and having visual coverage of surrounding walls of the second room, and second additional data captured by the mobile computing device in the second area; determining, by the computing device and using a first position of the mobile computing device in the first room based at least in part on the first additional data, a second position of the camera device in the first room relative to the first position of the mobile computing device, and determining, using the second position, a position of the first panoramic image within a first shape of the first room, the first shape being based at least in part on the visual coverage of the first panoramic image; determining, by the computing device and using a third position of the mobile computing device in the second room based at least in part on the second additional data, a fourth position of the camera device in the second room relative to the third position of the mobile computing device, and determining a position of the second panoramic image within a second shape of the second room using the fourth position, the second shape being based at least in part on the visual coverage of the second panoramic image; generating, by the computing device and using the first shape of the first room and the second shape of the second room and the determined positions of the first panoramic image and the second panoramic image, at least a partial floor plan of the building, including determining a relative position of the first shape of the first room and the second shape of the second room using the movement data of the mobile computing device; as well as The generated at least partial floor plan of the building with a visual indication overlay is displayed by the computing device to show the determined positions of the first panoramic image and the second panoramic image in the respective first room and the second room.

2. The computer-implemented method of claim 1, wherein: The computing device is the mobile computing device, wherein displaying the generated at least partial floor plan comprises displaying the generated at least partial floor plan to a user of the mobile computing device in the building, and wherein the method further comprises, after displaying the generated at least partial floor plan: receiving, by the mobile computing device from the user, an indication of a third room of the building separate from the first room and the second room; obtaining, by the mobile computing device, a third panoramic image captured by the camera device in the third room, and third additional data captured by the mobile computing device in the third room and comprising motion data from one or more inertial measurement unit sensors of the mobile computing device and visual data from one or more image sensors of the mobile computing device; determining, by the mobile computing device, a position of the third panoramic image within a third shape of the third room, the third shape being determined based on a combination of visual data of the third panoramic image and the third additional data captured in the third room; updating, by the mobile computing device, the at least partial floor plan to include a position of the third shape of the third room relative to at least one of the first shape of the first room and the second shape of the second room; and The updated at least partial floor plan of the building is displayed by the mobile computing device.

3. The computer-implemented method of claim 2, further comprising, prior to generating the at least partial floor plan: The first shape of the first room is determined by the mobile computing device based on a combination of visual data of the first panoramic image captured by the camera device and based on the first additional data captured by the mobile computing device, wherein The first shape is a first three-dimensional shape; determining, by the mobile computing device, the second shape of the second room based on a combination of visual data of the second panoramic image captured by the camera device and based on the second additional data captured by the mobile computing device, wherein the second shape is a second three-dimensional shape; and determining, by the mobile computing device, the third shape of the third room based on a combination of the visual data of the third panoramic image captured by the camera device and based on the third additional data captured by the mobile computing device, wherein the third shape is a third three-dimensional shape, And wherein generating the at least partial floor plan of the building includes positioning, by the mobile computing device, the first three-dimensional shape and the second three-dimensional shape relative to each other in the generated at least partial floor plan.

4. The computer-implemented method of claim 1, wherein: The first panoramic image captured by the camera device in the first area includes visual data of some or all of the first room, wherein the first additional data captured by the mobile computing device in the first area includes additional visual data of at least some of the first room, and wherein the method further comprises using at least one of the following operations (a) to (g): (a) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the additional visual data to identify the camera device in the additional visual data, and using the position of the identified camera device in the additional visual data as part of determining the second position of the camera device in the first room; or (b) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the additional visual data to identify one or more markers located on the camera device in the additional visual data, and using one or more positions of the identified one or more markers in the additional visual data as part of determining the second position of the camera device in the first room; or (c) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the visual data of the first panoramic image to generate a first estimated shape of the room, analyzing the additional visual data to generate a second estimated shape of the room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape at which the visual data of the first panoramic image was captured; or (d) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the visual data of the first panoramic image to generate a first estimated shape of the room, analyzing further depth data captured by the mobile computing device in the first room to generate a second estimated shape of the room, and comparing the first estimated shape and the second estimated shape to identify a location within the second estimated shape at which the visual data of the first panoramic image was captured; or (e) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing the visual data and the additional visual data of the first panoramic image to identify features visible in both the visual data and the additional visual data, and using the positions of the identified features as part of determining the second position of the camera device in the first room; or (f) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing movement data of the mobile computing device in the room as the mobile computing device moves to and / or from the position at which the camera device captured the first panoramic image to identify the position, and using the identified position as part of determining the second position of the camera device in the first room; or (g) determining, by the computing device, the second position of the camera device in the first room relative to the first position of the mobile computing device in the first room, comprising analyzing visual data of the first panoramic image and further visual data of another panoramic image captured in the first room to identify features visible in both the first panoramic image and the other panoramic image, and using the positions of the identified features to determine a common coordinate system for the first panoramic image and the other panoramic image for combining the visual data and the further visual data and as part of determining the second position of the camera device in the first room.

5. The computer-implemented method of claim 4, wherein: Using at least one of the operations (a) to (g) includes: the computing device combining information from multiple operations of the operations (a) to (g) by using at least one of confidence information generated for the multiple analysis operations or statistical analyses performed on the multiple analysis operations.

6. The computer-implemented method of claim 1 , further comprising at least one of: generating, performed by the computing device, the at least partial floor plan based at least in part on motion data obtained in the first room using one or more inertial measurement unit sensors of the mobile computing device and visual data obtained in the first room using one or more image sensors of the mobile computing device, including analyzing the motion data and the visual data to estimate a three-dimensional shape of the first room, analyzing additional visual data of the first panoramic image to identify one or more structural wall elements of a wall of the first room, generating one or more locations of the identified one or more structural wall elements of the wall of the first room within the estimated three-dimensional shape of the first room, and using the generated one or more locations as part of generating the at least partial floor plan; or generating, performed by the computing device, the at least partial floor plan based at least in part on a plurality of panoramic images obtained at a plurality of acquisition locations, including generating positions of the plurality of panoramic images relative to each other by determining directions between pairs of the plurality of panoramic images based on at least one of analyzing visual data of the plurality of panoramic images or analyzing movement data from the one or more inertial measurement unit sensors as the mobile computing device moves between acquisition locations of the plurality of panoramic images; or Performing generation of the at least partial floor plan based at least in part on a combination of operations by the mobile computing device and at least one additional computing device separate from the mobile computing device includes generating, by the mobile computing device, an initial estimated shape of the first room based on at least one of additional visual data of the first panoramic image or visual data captured by the mobile computing device in the first room, and including using, by the at least one additional computing device, the first shape of the first room to improve the initial estimated shape of the first room based at least in part on performing further analysis of the additional data.

7. A non-transitory computer-readable medium having stored thereon software instructions for causing one or more computing devices to perform automated operations, the automated operations comprising at least: obtaining, by the one or more computing devices, one or more panoramic images having visual data captured by a camera device in one or more rooms of a building and one or more additional images having additional visual data captured in the one or more rooms by a mobile computing device separate from the camera device; determining, by the one or more computing devices and for each of the one or more rooms, a shape of the room by using at least one of the visual data of the one or more panoramic images captured by the camera device or the additional visual data of the one or more additional images captured by the mobile computing device; By the one or more computing devices and for each of the one or more rooms: determining a first location of the mobile computing device in the room in which the at least one additional image was captured based at least in part on the additional visual data of at least one of the additional images, and determining a second position of the camera device in the room for capturing at least one of the panoramic images, wherein the second position of the camera device in the room is determined relative to the first position of the mobile computing device in the room; generating, by the one or more computing devices, mapping information of the one or more rooms, the mapping information including, for each of the one or more rooms, a position of the at least one panoramic image captured in the room within the determined shape of the room, including using the second position of the camera device in the room as part of determining the position of the at least one panoramic image captured in the room; and The generated mapping information of the one or more rooms is provided by the one or more computing devices to enable display of the determined shape of each of the one or more rooms with one or more superimposed visual indications to show the location of the at least one panoramic image in each of the one or more rooms.

8. The non-transitory computer readable medium of claim 7, wherein: Obtaining the one or more panoramic images includes obtaining, by the camera device, a plurality of panoramic images, the plurality of panoramic images including at least one panoramic image of each of two or more rooms of the building, wherein obtaining the one or more additional images includes obtaining, by the mobile computing device, a plurality of additional images, the plurality of additional images including at least one additional image of each of the two or more rooms, and wherein generating the mapping information includes generating, by the one or more computing devices, at least a partial floor plan of the building, the at least partial floor plan including determined shapes of the two or more rooms positioned relative to each other.

9. The non-transitory computer readable medium of claim 8, wherein: The mobile computing device is the one or more computing devices that perform the automated operations, and wherein the stored software instructions, when executed by the mobile computing device, cause the mobile computing device to perform further automated operations, the further automated operations comprising at least: capturing, by the mobile computing device, movement data during travel between the two or more rooms; and performing, by the mobile computing device, positioning the determined shapes of the two or more rooms relative to each other based at least in part on the captured movement data, And wherein providing the generated mapping information includes: displaying, by the mobile computing device, the generated at least partial floor plan of the building to a user to enable identification of additional areas of the building for acquisition of additional panoramic images.

10. The non-transitory computer readable medium of claim 7, wherein: Generating the mapping information includes generating an at least partial floor plan of the building having representations of at least two rooms of the building, wherein providing the generated mapping information includes displaying, by the one or more computing devices, the generated at least partial floor plan to a user, and wherein the automated operation further includes, after displaying the generated at least partial floor plan: receiving, by the one or more computing devices from the user, an indication of an additional room of the building separate from the at least two rooms; obtaining, by the one or more computing devices, further panoramic images captured by the camera device in the additional room, and further additional images captured by the mobile computing device in the additional room, and further motion data captured from one or more inertial measurement unit sensors of the mobile computing device in the additional room; determining, by the one or more computing devices, a position of the further panoramic image within a shape of the additional room, the shape of the additional room being determined based on a combination of the visual data of the further panoramic image and the additional visual data of the further additional image and the further motion data captured in the additional room; updating, by the one or more computing devices, the at least partial floor plan to position the shape of the additional room relative to at least one of the determined shapes of the at least two rooms; and The updated at least partial floor plan of the building is displayed by the one or more computing devices.

11. The non-transitory computer readable medium of claim 7, wherein: The one or more rooms include at least a first room and a second room, wherein the automated operation further includes: generating, by the one or more computing devices, a first three-dimensional shape of the first room based on a combination of visual data of the at least one panoramic image captured by the camera device in the first room and the additional visual data based on the at least one additional image captured by the mobile computing device in the first room, including determining at least one of a three-dimensional point cloud modeling at least some walls of the first room or a planar surface representing the at least some walls of the first room, and also including identifying a doorway between the first room and the second room; generating, by the one or more computing devices, a second three-dimensional shape of the second room based on a combination of visual data of the at least one panoramic image captured by the camera device in the second room and the additional visual data based on the at least one additional image captured by the mobile computing device in the second room; and Generating, by the one or more computing devices, at least a partial floor plan of the building, includes positioning the first three-dimensional shape and the second three-dimensional shape relative to each other in the generated at least partial floor plan based at least in part on the location of the determined doorway between the first room and the second room.

12. The non-transitory computer readable medium of claim 7, wherein: The mobile computing device is the one or more computing devices, and wherein determining the shape of each of the one or more rooms also includes: determining, by the mobile computing device, a three-dimensional shape of the room based on a combination of the visual data of the at least one panoramic image captured by the camera device in the room and the additional visual data of the at least one additional image captured by the mobile computing device in the room.

13. The non-transitory computer readable medium of claim 12, wherein: The one or more panoramic images each include 360 ​​degrees of horizontal visual coverage about a vertical axis, and wherein determining the three-dimensional shape of each of the one or more rooms is performed without using any depth information from any depth sensing sensor for determining distances from a position of the camera device or the mobile computing device in the one or more rooms to surrounding surfaces.

14. The non-transitory computer readable medium of claim 7, wherein: The automated operation also includes obtaining, by the mobile computing device, additional motion data captured by one or more inertial measurement units of the mobile computing device in the one or more rooms, and wherein determining the shape of each of the one or more rooms is further based in part on the additional motion data.

15. The non-transitory computer readable medium of claim 7, wherein: The stored software instructions, when executed by the one or more computing devices, cause the one or more computing devices to perform further automated operations, the further automated operations comprising using at least one of the following operations (a) to (g): (a) determining, by the one or more computing devices, a second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the additional visual data of the at least one additional image captured by the mobile computing device in the room to identify the camera device in the additional visual data, and using the position of the identified camera device in the additional visual data as part of determining the second position of the camera device in the room; or (b) determining, by the one or more computing devices, a second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the additional visual data of the at least one additional image captured by the mobile computing device in the room to identify one or more markers in the additional visual data located on the camera device, and using one or more positions of the identified one or more markers in the additional visual data as part of determining the second position of the camera device in the room; or (c) determining, by the one or more computing devices, a second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the visual data of the at least one panoramic image captured by the camera device in the room to generate a first estimated shape of the room, analyzing the additional visual data of the at least one additional image captured by the mobile computing device in the room to generate a second estimated shape of the room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape of the visual data capturing the at least one panoramic image in the room; or (d) determining, by the one or more computing devices, a second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the visual data of the at least one panoramic image captured by the camera device in the room to generate a first estimated shape of the room, analyzing further depth data captured by the mobile computing device in the room to generate a second estimated shape of the room, and comparing the first estimated shape to the second estimated shape to identify a location within the second estimated shape of the visual data capturing the at least one panoramic image in the room; or (e) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing the visual data of the at least one panoramic image captured by the camera device in the room and the additional visual data of the at least one additional image captured by the mobile computing device in the room to identify features visible in both the visual data and the additional visual data, and using the positions of the identified features as part of determining the second position of the camera device in the room; or (f) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, comprising analyzing movement data of the mobile computing device in the room as the mobile computing device moves to and / or from the position where the camera device captured the at least one panoramic image in the room to identify the position, and using the identified position as part of determining the second position of the camera device in the room; or (g) determining, by the one or more computing devices, the second position of the camera device in each of the one or more rooms relative to the first position of the mobile computing device in the room, including analyzing visual data of two or more panoramic images captured in the room to identify features visible in both of the two or more panoramic images, and using the positions of the identified features to determine a common coordinate system for the two or more panoramic images and as part of determining the second position of the camera device in the room.

16. The non-transitory computer readable medium of claim 15, wherein: For each of the one or more rooms, using at least one of the operations (a) to (g) includes: the one or more computing devices combining information from multiple operations of the operations (a) to (g) for the room by using at least one of confidence information generated for the multiple analysis operations or statistical analyses performed for the multiple analysis operations.

17. The non-transitory computer readable medium of claim 15, wherein: Determining, for each of the one or more rooms, the second position of the camera device in the room relative to the first position of the mobile computing device in the room further includes: determining the orientation of the camera device in each of the one or more rooms during capture of each of the at least one panoramic image in the room, wherein generating the mapping information is further based in part on the determined orientation for each of the one or more rooms and each of the at least one panoramic image captured in the room, and wherein using at least one of the operations (a) to (g) for each of the one or more rooms includes combining, by the one or more computing devices, information from all of the operations (a) to (g) for the room.

18. The non-transitory computer readable medium of claim 7, wherein: The automated operation also includes at least one of the following: performing, by the one or more computing devices, generating the mapping information based at least in part on motion data obtained using one or more inertial measurement unit sensors of the mobile computing device in the one or more rooms, including analyzing the additional visual data of the at least one additional image captured by the mobile computing device in one of the one or more rooms and further motion data captured by the mobile computing device in the one room to estimate a three-dimensional shape of the one room, analyzing the visual data of the at least one panoramic image captured in the one room to identify one or more structural wall elements of a wall of the one room, generating locations of the identified one or more structural wall elements of the wall of the one room within the estimated three-dimensional shape of the one room, and using the generated locations as part of generating the mapping information; or performing, by the one or more computing devices, generating the mapping information based at least in part on a plurality of panoramic images obtained at a plurality of acquisition locations, including generating positions of the plurality of panoramic images relative to each other by determining directions between pairs of the plurality of panoramic images based on at least one of analyzing visual data of the plurality of panoramic images or analyzing movement data from the one or more inertial measurement unit sensors as the mobile computing device moves between acquisition locations of the plurality of panoramic images; or Performing generation of the mapping information based at least in part on a combination of operations by the mobile computing device and at least one of the one or more computing devices separate from the mobile computing device includes generating, by the mobile computing device, an initial estimated shape of the one of the one or more rooms based at least in part on at least one of the visual data of the at least one panoramic image captured by the camera device in one of the one or more rooms or the additional visual data of the at least one additional image captured by the mobile computing device in the one room, and using, by the at least one computing device separate from the mobile computing device, the determined shape of the one room to improve the initial estimated shape of the one room based at least in part on performing further analysis on the additional data.

19. A system comprising: one or more hardware processors of one or more computing devices; as well as one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations, the automated operations comprising at least: obtaining a plurality of images having visual data captured by a camera device in two or more rooms of a building and a plurality of additional images having additional visual data captured in the two or more rooms by a mobile computing device separate from the camera device; determining a shape of each of the two or more rooms using at least one of the visual data of the plurality of images captured by the camera device or the additional visual data of the plurality of additional images captured by the mobile computing device; For each of the two or more rooms: determining, during capturing of one or more of the additional images in the room, a first location of the mobile computing device in the room based at least in part on the additional visual data of the one or more additional images, and During capturing of at least one of the plurality of images in the room, determining a second position of the camera device in the room, wherein the second position in the room is determined relative to the first position in the room; generating mapping information including the determined shape of each of the two or more rooms positioned relative to each other and, for each of the two or more rooms, a position of the at least one image captured in the room within the determined shape of the room, including using the second position of the camera device in the room as part of determining the position of the at least one image of the room; and The generated mapping information of the two or more rooms is provided to enable display of the generated mapping information.

20. The system of claim 19, in, Generating the surveying and mapping information includes: generating at least a partial floor plan of the building, wherein the mobile computing device is the one or more computing devices that perform the automated operations, and wherein the stored instructions include software instructions that, when executed by the mobile computing device, cause the mobile computing device to perform further automated operations, the further automated operations comprising at least: capturing movement data as the mobile computing device moves between the two or more rooms, and performing positioning of the determined shapes of the two or more rooms relative to each other based at least in part on the captured movement data, and Wherein providing the generated mapping information comprises: displaying the generated at least partial floor plan of the building to a user so that additional areas of the building can be identified for capturing additional images.

Images