Automatically generating building plan with associated absolute values using coordinated multiple data sets

Abstract

Techniques are described for using a data capture device at a building to automatically generate a building plan view and determine associated absolute location data for the generated plan view, such as by associating individually captured GPS data or other absolute location data with the plan view. In some cases, a building plan is automatically generated by analyzing visual data of images captured by a camera device at a plurality of image acquisition sites to determine room shapes of surrounding rooms, and associating the GPS absolute location data with the generated plan graph using additional data captured at other data capture locations of the building by a separate mobile device moving independently of the camera device, an image acquisition location and its surrounding room shape, such as by extending absolute location data from a mobile device to a camera device.

Description

Technical Field

[0001] The following disclosure generally relates to techniques for automatically generating a building floor plan; determining an absolute location associated with the building floor plan based at least in part on mapping images captured within the building to location data captured separately from the images; and subsequently using the generated building floor plan information in one or more ways to improve navigation of the building. Background Art

[0002] In various fields and environments, such as building analysis, property inspections, real estate acquisition and development, remodeling and improvement services, general contracting, and other environments, it may be desirable to view information about the interior of a house, office, or other building without having to physically travel to and enter the building, including determining actual construction information about the building, rather than design information from before the building was built. However, it may be difficult to effectively capture, represent, and use such building interior information, including displaying visual information captured inside the building to a user at a remote location (e.g., enabling the user to fully understand the layout and other details of the interior, including controlling the display in a manner selected by the user). In addition, although a floor plan of a building may provide some information about the layout and other details of the interior of the building, such use of the floor plan has some drawbacks in some cases, including that the floor plan is difficult to construct and maintain, difficult to accurately scale and populate with information about the interior of a room, difficult to visualize and otherwise use (including about its surroundings), and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 Included are diagrams illustrating one or more exemplary building interior environments and computing systems used in embodiments of the present disclosure, including generating and presenting information representative of a building interior.

[0004] Figure 2A to Figure 2-O and Figure 2P to Figure 2V An example of automated operations for generating and presenting floor plan information of a building based on images taken inside the building and additional capture data is shown, so as to automatically determine and use the absolute acquisition locations of such images.

[0005] Figure 3 is a block diagram illustrating a computing system suitable for executing an embodiment of a system that executes at least some of the techniques described in this disclosure.

[0006] FIG. 4A to FIG. 4BAn exemplary implementation of a flow chart for a Building Floor Plan Generator and Location Determiner and Presenter (BFPGLDP) system routine according to an embodiment of the present disclosure is shown.

[0007] Figure 5 An exemplary embodiment of a flow chart of an Image / Data Capture and Analysis (IDCA) system routine is shown according to an embodiment of the present disclosure.

[0008] FIG. 6A to FIG. 6B An exemplary implementation of a flow chart for a Mapping Information Generation Manager (MIGM) system routine according to an embodiment of the present disclosure is shown.

[0009] FIG. 7A to FIG. 7B An exemplary implementation of a flow chart for a building information viewer system routine is shown in accordance with an implementation of the present disclosure. DETAILED DESCRIPTION

[0010] The present disclosure describes techniques for using a computing device to perform automated operations involving automatically generating a building floor plan using visual data of an image within a building, determining an associated absolute position of the generated floor plan based at least in part on mapping the image to absolute position data captured separately from the image, then extending the absolute position data associated with the image to surrounding building floor plan structural elements visible in the image, presenting the building floor plan using surrounding reality data of the associated absolute position of the building floor plan, and in some cases, subsequently using the generated building floor plan information in one or more additional ways to further improve navigation of the building and / or its surroundings. In at least some embodiments, the described techniques include: automatically generating a building floor plan based at least in part on analyzing visual data of images captured by a camera device at multiple image acquisition locations in a building to determine room shapes of rooms surrounding the image acquisition locations; and automatically determining GPS (Global Positioning System) location data or other absolute location data and associating it with the generated floor plan based at least in part on additional data captured at other data acquisition locations at the building (e.g., additional data captured at other data acquisition locations at the building). The absolute location data is extended from the other data capture locations to the image acquisition location of the camera device by a separate mobile data capture device that moves independently of the camera device, for example. and associated surrounding room shapes. The image may, for example, include a panoramic image or other image (e.g., a rectilinear perspective image) acquired by one or more camera devices (generally referred to herein as a "target image") at an image acquisition location within or around a multi-room building (e.g., a house, office, etc.). The following includes additional details regarding automatically determining and using acquisition location information of a building image to determine the associated absolute location of a resulting generated floor plan of a building, as well as presentation and other uses of the resulting generated floor plan, and in at least some embodiments, some or all of the techniques described herein may be performed via automatic operation of a building floor plan generator and position determiner and renderer ("BFPGLDP") system. As further discussed below.

[0011] As described above, target images of a building and / or other acquired data can be analyzed to generate a plan model and / or other mapping information of the building (e.g., a two-dimensional model of the interior of a building without wall height data, such as an orthographic projection or a top view; a three-dimensional model of the interior of a building; a linked group of target images with directional information between pairs of images, etc.) In at least some embodiments, structural elements such as walls, doorways and windows, and non-doorway wall openings are identified, for example by using visual data and determined acquisition positions of acquired images to determine the relative position of the acquisition position of each image to such identified structural elements (e.g., within a local coordinate system for the image), to determine room shapes based on the identified structural elements, and to identify the acquisition position of each image within one of the room shapes (e.g., within one of the room shapes). In at least some such embodiments, the automatic analysis and use of the captured images and / or other data is further performed without or using any captured depth data from any depth sensor or other distance measuring device regarding the distance from the capture location to walls or other objects in the surrounding building, while in other embodiments, such depth data may be captured and used. In various embodiments, such generated floor plans and / or other mapping information may be further used in various ways, such as for controlling navigation of a mobile device (e.g., an autonomous vehicle), for display or other presentation on one or more client devices in a corresponding GUI (graphical user interface), etc. The following includes additional details related to generating a floor plan of a building using captured images of the building and optionally using other captured data of the building.

[0012] As described above, automatic operation of the BFPGLDP system may include automatically mapping a target image (e.g., a target panoramic image) acquired in a building (e.g., in one or more rooms or other defined areas) to other absolute position data acquired at the building separate from image acquisition, and using such mapping to determine the associated absolute position of a floor plan generated from the target image. For example, when acquiring one or more target images, GPS position data or other absolute position data acquired at one or more other data capture locations at the building is enabled to be extended to the image acquisition location of the camera device, and further extended to other locations around the image acquisition location, the other locations being determined at least in part based on an analysis of visual data of the one or more target images (e.g., the location of the room shape of the surrounding rooms, for example, the location of at least the walls of the room). Absolute location data for a data capture location may have various forms and may be determined in various ways in various embodiments, with non-exclusive examples including obtaining absolute GPS location data for a capture device from one or more GPS receivers on the capture device and / or receiving or determining absolute location data in other ways (e.g., via other wireless transmissions such as Bluetooth, NFC, etc.); via determined relative positions to other nearby objects with known absolute locations; using motion data from a capture device at a known location, such as from one or more IMU (inertial measurement unit) sensors on the capture device, etc.), and in some embodiments and scenarios (e.g., obtaining and using geographic direction data from a compass sensor on the capture device). Although mobile capture devices are referred to herein in the singular, it will be appreciated that multiple mobile capture devices may be used in some embodiments and scenarios of a given building, such as different mobile devices capturing absolute location data at different times (e.g., during different data capture sessions) and / or at different data capture locations (in the same or different rooms or other defined areas as one or more other mobile capture devices) and / or of different types. Furthermore, in various embodiments, such a mobile capture device may take various forms, including as a mobile computing device (e.g., a smartphone, tablet, or laptop computer, etc.) that includes computing capabilities and may be used to perform at least some automated operations.

[0013] In various embodiments, determining the location of an image acquisition location at which an image acquisition device acquires one or more target images based on other absolute location data of a mobile capture device can be performed in various ways. In at least some embodiments and scenarios, GPS data or other absolute location data is acquired by a mobile capture device at multiple times (e.g., periodically, substantially continuously, etc.), the mobile capture device being separated from a camera or other image acquisition device that captures one or more target images at the image acquisition location but may be near the image acquisition device (e.g., in the same room, carried by the same user, within a threshold distance, etc.) - in other embodiments and scenarios, some or all of such absolute location data may be obtained by the image acquisition device, but separate from the image acquisition activity and not directly associated with the acquired target image when acquired. In addition, for each of some or all of such captured target images, the image acquisition device and / or other related analysis devices may provide additional data, such as providing a linear stream of image acquisition events in some embodiments and scenarios - non-exclusive examples of other data that may optionally be associated with each of some or all of the events, and their associated target images include posture data for the image acquisition device and the corresponding resulting target image acquired for the event. objects detected in visual data of the target image, one or more types of metadata used for acquisition of the target image (e.g., model and / or type of image acquisition device, type and / or version of software used on the image acquisition device, etc.), operator user actions associated with acquisition of the target image, location of the target image within a room shape or within a floor plan in a local coordinate system for the room shape or floor plan, etc. In some embodiments and scenarios, mapping the GPS data and / or other absolute position data to the acquired target image includes determining a time associated with image acquisition (e.g., based on event data associated with image acquisition), and selecting or otherwise determining a GPS data point or other absolute position data point to initially inaccurately represent the acquisition location of the image—such determination of a representative absolute position data point may, for example, include determining a centroid or other average of a plurality of absolute position data points obtained in a time window around the image acquisition time (optionally weighted by a time difference from the image acquisition time and / or by one or more other attributes), or selecting a single best representative absolute position data point from a plurality of absolute position data points obtained in a time window around the image acquisition time (e.g., obtained in a time window around the image acquisition time). Having representative absolute position data points for the most recent times before and / or after the image acquisition time, etc.After each of the multiple target images captured at multiple acquisition locations of the building is initially associated with a GPS data point or other absolute position data point, the technology may include further estimating a global best fit transformation between the set of acquisition locations of the target images as represented in a common local coordinate system and the generated floor plan and its associated determined absolute position data points, such as by converting each absolute position data point to a position in the local coordinate system of the floor plan. And executing one or more algorithms to determine the best matching rigid transformation between pairs of absolute position data points and local coordinate system data points, such as using a Kabsch algorithm and / or a Quaternion estimator algorithm and / or another algorithm that solves the Wahba problem - if compass heading information is further available for the captured target images and / or absolute position data points, the technology may further use heading information to separately estimate the orientation of the target images and the global orientation of the floor plan. In some embodiments and situations, the determination of the global transformation includes at least the following:.

[0014] - Convert each representative GPS data point having the (latitude, longitude) format to the (X,Y) format in the local coordinate system of the plan view;

[0015] N is the number of target images. We then have N pairs {(X_i,Y_i),(x_i,y_i)}, i=1, etc., N, where (X_i,Y_i) is the estimated target image location in the local coordinate system of the planar image, and (x_i,y_1) is the absolute position data point (e.g., latitude, longitude) associated with the estimated target image location. And

[0016] - Estimate the rigid transformation using N pairs of {(X_i,R1),(_i,H1)} by one or more of several methods, such as those discussed above.

[0017] In addition, in some embodiments and scenarios, the determination of such a rigid transformation or other global transformation between the image acquisition positions in local coordinates for the plan view and the associated absolute position data points may include additional operations. For example, the acquired GPS data points or other absolute position data points may be processed before they are initially associated with the target image, such as by performing a smoothing operation to reduce noisy data (e.g., eliminating or discounting outliers). In addition, after initially determining the absolute position data points to be associated with the target image and after performing any optional smoothing or other initial processing, additional processing may be performed to refine the resulting initial rigid transformation, such as using an iterative closest point (ICP) or other refinement technique to improve the rigid transformation determination, for example, doing so may include preparing two sets of position data points (e.g., for example,). A first set of GPS absolute position data points and a second set of estimated target image acquisition position data points, the first set of GPS absolute position data points being captured near the acquisition time of the target image, such as within a time window, and the second set of estimated target image acquisition position data points being within a local coordinate system for a floor plan of a building, such as by applying an initial rigid transformation to each target image prior to refinement), matching each local coordinate position data point in the second set to the nearest absolute position data point in the first set; estimating a rigid transformation between the matched data points, iterating the matching and estimating operations to further refine the rigid transformation until one or more termination criteria are met.Furthermore, whether in addition to or in lieu of smoothing or other initial processing and / or refinement, other additional operations may include one or more of the following operations: performing a point group registration technique using the first and second sets of position data points as described above to find an optimal global transformation; using various types of data as input to a trained deep learning system that outputs a global transformation, such as input data including some or all of the types of data discussed above with respect to initial rigid transformation determination as well as additional data, such as overall floor shape, room shape, locations of doorways and / or windows and / or one or more of non-doorway wall openings, etc.; using detected motion trajectory information for a separate data capture device to align the captured absolute position data with a doorway and / or exterior structure of a building, such as based on identifying a transition from outside the building to inside the building (e.g., from an operator's position at the time of the transition based on analysis of the captured visual data); input) to determine the absolute position of an exterior doorway, etc.) and select the associated absolute position data at that time to detect linear exterior motion and align it with an exterior walkway or driveway, etc.; use additional human operator activity data to improve GPS signal quality, such as having a human operator user explicitly capture GPS absolute position data at an exterior doorway to have a better baseline for interior locations that may have lower accuracy GPS data, having a human operator user track some or all of the exterior boundary of a building so that the shape of a generated floor plan matches that boundary, having a human operator user track a specified exterior pattern prior to entering the building to improve GPS signal calibration for subsequent establishment of GPS data points, etc.; use input from one or more human operators to provide an initial estimate of the absolute data location of some or all of the target images and / or an initial estimate of an initial rigid transform, and / or provide refinement of the determined rigid transform and / or provide refinement of the determined absolute position data of the floor plan, etc. The following includes additional details regarding mapping images acquired at a building to absolute position data captured at the building separately from the images, as well as the additional data captured and the type of mobile capture device.

[0018] As described above, the automatic operation of the BFPGLDP system may also include automatically presenting a building floor plan with associated absolute position data using surrounding reality data of the associated absolute position of the building floor plan. As a non-exclusive example, in these embodiments, as the zoom level increases and the available space on the screen for the floor plan allows, a map of the surrounding area of ​​a building with a generated floor plan with associated absolute position data can be displayed at various zoom levels, wherein a 2D (two-dimensional) or 3D (three-dimensional) floor plan model of the building shown on the map is displayed at the absolute position determined for the floor plan. Various interior aspects of the building can be displayed on the floor plan, such as walls, doorways, non-doorway wall openings, stairs, windows, fixtures (e.g., lights and plumbing fixtures), built-in and installed elements (e.g., islands, fireplaces, cabinets, cabinets, etc.), appliances, furniture, etc. Included are displaying different types of interior aspects at different zoom levels (e.g., aspects of the structure at only one zoom level, adding one or more types of additional interior aspects at each of one or more additional zoom levels, etc.), and including optionally using different colors, patterns, icons, bounding boxes, or other visual elements to emphasize or de-emphasize particular aspects of interest. Additionally, if the building is part of a larger structure (e.g., a unit in a multi-unit structure) and / or has associated nearby structures (e.g., a detached garage, shed, detached accessory dwelling unit, etc.), the plan model is displayed relative to the other structures or structure portions to reflect those realistic relationships. In some embodiments, such a 2D or 3D plan model is displayed on the map in place of any other visual representation of the building, while in other embodiments, the plan model is overlaid on top of an image or other visual representation of the building (e.g., to fit the plan model to the exterior boundaries of the building, as visible in an overhead or street-level image of the building that is part of or overlaid on the map). Furthermore, when a building has multiple stories, information from the multiple stories can be presented in various ways, such as displaying interior aspects of different stories simultaneously but using different visual aspects to distinguish them (e.g., different colors, patterns, etc.), displaying different stories sequentially (e.g., automatically, such as a fixed amount of time for each story; as indicated by manual instructions, etc., or as selected by a user), displaying (or highlighting) different stories at different zoom levels (e.g., displaying the top story at an initial zoom level, and exposing lower stories as the zoom level increases), etc. Furthermore, such displayed maps can be interactive in at least some embodiments and situations, for example, via GUI (graphical user interface) operations, such as via mouse and / or keyboard and / or touch screen input, including, for example, finger press actions, to implement zooming and / or scrolling operations.Additionally, various details about the surrounding area outside of the floor plan of the building may be displayed on the map in various ways, such as highlighting one or more types of neighborhood information or other nearby information (e.g., including pointers or other directional information for external locations such as schools, hospitals, roads, parks, etc.) - in some embodiments and scenarios, some or all of the neighborhood / nearby information may be overlaid on the visual representation of the floor plan map or otherwise included on the visual representation of the floor plan, as well as including information about the visual representation of the floor plan, such as the location of adjacent roads, trees, other buildings, etc. (e.g., based on information extracted from the map or otherwise available, such as information extracted from a public data source or other data source) to provide a "field plan" visualization. Additionally, in some embodiments, the type and context of the additional information displayed may vary with zoom level and / or based on user selections or preferences, and other types of information from external surrounding locations may similarly be added to the floor model and / or its visual representation (e.g., as displayed information visible through a window, with directional information within the floor model to a particular external surrounding location, etc.). A compass or other indication of geographic north and / or one or more other directions may similarly be added to a plan model and / or visual representation thereof, etc. The following includes additional details regarding the use of surrounding real-world data to present a building floor plan with associated absolute position data.

[0019] The described techniques provide various benefits in various embodiments, including allowing partial or complete floor plans of multi-room buildings and other structures to be automatically generated simultaneously with the acquisition of one or more target images acquired for the building or other structure, and / or allowing such partial or complete floor plans to be augmented with information regarding associated absolute position data, including in some embodiments without having or using information from a depth sensor or other distance measuring device regarding the distance of the image's acquisition location to walls or other objects in the surrounding building or other structure. Non-exclusive examples of such benefits include the following: the ability to provide feedback to an operator of the camera device during capture of one or more target images acquired for a building or other structure (e.g., displaying or otherwise providing to the operator user a determined room shape surrounding a room, such as a portion of a partial or complete floor plan of a building or other structure; enabling movement to different image acquisition positions that provide improved visual data, such as moving near a window to obtain visual data of objects outside the building having known position data, etc.), including optionally allowing the user to determine one or more other areas of the building at which to acquire one or more additional target images (e.g., for a partial floor plan, acquiring additional target images in other areas of the building that are not yet represented in the partial floor plan), and, for example, in a near-time or real-time manner related to the acquisition of the target images (e.g., for a partial floor plan, acquiring additional target images in other areas of the building). Utilizing initial information determined on a mobile computing device for immediate use, such as for display to a user on the mobile computing device, while also being provided to one or more other computing devices, such as a remote server computing system, for refinement or other updating); the ability to provide other types of feedback to a user of the mobile device or other conveyance, such as moving to a different data capture location that provides improved additional data (e.g., moving near a window or outside a doorway to obtain a stronger GPS signal and / or visual data of an object outside a building with known position data; moving to a specified area near a camera device, such as within the visible range of the camera device, or a specified area of ​​that range, such as centered about the visible coverage of the camera device at the specified range so that the mobile device or its conveyance is visible to the camera device, such as for determining inter-position pose data between the image acquisition location of the camera device and the data capture location of the mobile device at the specified area, etc.); etc. Furthermore, the described automated techniques allow the generation of such floor plans and the association with absolute position data to be determined more quickly than previously available techniques, and in at least some embodiments, with greater accuracy, including by using information obtained from the actual building environment (rather than from plans of how the building should theoretically be constructed), and enabling the capture of changes to structural elements that occur after the building is initially constructed.Such described techniques also provide the benefit of allowing a mobile device (e.g., a semi-autonomous or fully autonomous vehicle) to perform improved automatic navigation of a building, including significantly reducing the computing power and time used to attempt to otherwise learn the layout and / or location of the building. In addition, in some embodiments, the described techniques can be used to provide an improved GUI in which a user can more accurately and quickly obtain information about the interior (e.g., for navigating the interior) and / or exterior and / or surrounding areas of a building, including in response to a search request, as part of providing personalized information to a user, as part of providing a user with value estimates and / or other information about a building. Etc. Various other benefits are also provided by the described techniques, some of which are further described elsewhere herein.

[0020] As described above, generating a partial or complete floor plan for a building may include: analyzing visual data of one or more target images captured by a camera device at one or more image acquisition locations of a room of the building (or other defined area at the building) to determine at least some of the walls of the room visible in the visual data, and combining multiple pieces of determined wall data to form a room shape for the surrounding rooms (or other shapes of other defined areas), for example, such determination of the walls may include: modeling the walls as a grouping of planes and / or 3D data points, and the resulting determined room shape may be a 3D (three-dimensional) and / or 2D (two-dimensional) room shape based at least in part on the walls and their inter-wall boundaries; and in at least some embodiments and scenarios, similarly modeling some or all of the floor and / or ceiling (for example, for a 3D room shape). For example, in at least some embodiments, the described techniques may include: using one or more trained neural networks or other techniques to estimate a 3D room shape shown in one or more such target images, as non-exclusive examples, such 3D room shape estimation may include one or more of the following: using a trained convolutional neural network or other analysis technique to take one or more target images as input and estimate 3D point clouds of walls and other surfaces of the enclosed room from the visual content of the target images, and / or estimating piecewise planar representations (e.g., 3D walls and other planar surfaces) of the enclosed room from the visual content of the one or more target images; using a trained neural network or other analysis technique to take one or more target images as input and estimate wireframe structure lines of the enclosed room based on the visual content of the target images; (e.g., structural lines showing one or more of the boundaries between walls, the boundaries between walls and ceilings, the boundaries between walls and floors, the outlines of doorways and / or other room-to-room wall openings, the outlines of windows, etc.); using a trained neural network or other analysis technique to detect wall structural elements in the visual content of one or more target images (e.g., windows and / or skylights; passages to enter and / or exit a room, such as doorways and other openings in walls, stairs, hallways, etc.; boundaries between adjacent walls; boundaries between walls and floors; boundaries between walls and ceilings; corners (or solid geometric vertices) where at least three surfaces or planes meet, etc.), and optionally detecting other fixed structural elements (e.g., countertops, bathtubs, sinks, islands, wall stoves, etc.), and optionally generating 3D bounding boxes for the detected elements, etc.Although camera devices are referred to herein in the singular, it will be appreciated that multiple camera devices may be used in some embodiments and scenarios for a given building, such as different camera devices that acquire different target images at different times (e.g., during different image acquisition sessions and / or at different image acquisition locations, whether in the same or different rooms or other defined areas as one or more other camera devices), different camera devices that acquire different target images at the same time (e.g., during the same image acquisition session and at different or the same image acquisition locations, whether in the same or different rooms or in other defined areas as one or more other camera devices), and the like.

[0021] In addition, in some embodiments and scenarios, analysis of visual data of one or more target images captured by one or more camera devices at one or more image acquisition locations in a room (or other defined area) may be combined with additional room shape data, wherein the additional room shape data is determined from analysis of other data captured by one or more mobile devices at one or more other data capture locations in the room (or other defined area), where non-exclusive examples include the following: analyzing additional visual data of additional images captured by the mobile device to determine information about at least some walls (and optionally, some or all of the floor and / or ceiling) of the surrounding room, optionally combined with IMU data to generate a 3D point cloud of at least some of the room shape; analyzing depth data captured by the mobile device using one or more sensors that measure depth or otherwise determine the distance to walls or other surrounding objects, etc. In at least some embodiments, operation of the mobile device may be based at least in part on performing SLAM (Simultaneous Localization And Mapping) and / or SFM (Structure from Motion) and / or MVS (multiple-view stereovision) analysis, such as by using motion data from an IMU sensor on the mobile computing device in combination with visual data from one or more image sensors on the mobile computing device, and in at least some such embodiments, including using additional data captured by the mobile computing device to generate an estimated three-dimensional (“3D”) shape of an enclosed room (e.g., based on a 3D point cloud having a plurality of 3D data points and / or estimated planar surfaces of walls and optionally a floor and / or ceiling), and in some such embodiments, these automatic operations are performed without using any depth sensors or other distance measuring devices regarding the distance from the mobile computing device to walls or other objects in the surrounding room, while in other embodiments, the mobile computing device (or other additional associated mobile devices) may capture depth data to the walls of the surrounding room and use the captured depth data as part of determining the position of the mobile computing device. Automatic determination of the location of the mobile computing device may be further performed in some embodiments as part of generating a path of travel of the mobile computing device through the enclosed room (e.g., using one or more of SLAM, SFM, and / or MVS analysis), whether instead of or in addition to generating a 3D shape of the enclosed room, in other embodiments, automatic determination of the location of the mobile computing device may be based at least in part on other analysis, such as through Wi-Fi triangulation, Visual Inertial Odometry (“VIO”), etc.Additional details related to determining room shapes and combining room shapes to form partial or complete building plans are included below.

[0022] As described above, in at least some embodiments, a building floor plan model having associated room shape information for some or all rooms of a building may be generated and used, and in various embodiments may have various forms, such as a 2D (two-dimensional) floor plan of a building (e.g., an orthographic top view or other top view of a schematic floor plan that does not include or display height information) and / or a 3D (three-dimensional) or 2.5D (two and a half dimensional) floor plan model of a building that displays height information. In addition, in some embodiments, a target image (and optionally additional images) may be acquired outside of one or more buildings, such as in one of a plurality of separate areas (e.g., for a house, garden, courtyard, deck, back yard, side yard, front yard, swimming pool, parking lot, dock, etc.) of one or more properties, each of which has a previously or concurrently determined area shape (e.g., 3D shape, 2D shape, etc.), and if so, the shape of the surrounding area of ​​the image may be similarly automatically determined using the techniques described herein and included as part of the building floor plan.

[0023] As described above, in at least some embodiments and scenarios, some or all of the target images acquired for a building may be panoramic images, each of which is acquired at one of a plurality of acquisition locations within or around the building, so as to generate a panoramic image at each such acquisition location from one or more videos captured at the acquisition location (e.g., 360° video acquired from a smartphone or other mobile device held by a user while rotating it at the acquisition location), or from a plurality of images captured from the acquisition location in multiple directions (e.g., from a smartphone or other mobile device held by a user while rotating it at the acquisition location; from automatic rotation of a device at the acquisition location, such as on a tripod at the acquisition location, etc.), or to simultaneously capture all image information at a particular acquisition location (e.g., using one or more fisheye lenses), etc. It should be understood that such panoramic images may in some cases be represented in a spherical coordinate system and provide up to 360° coverage about the horizontal and / or vertical axes (e.g., 360° coverage along the horizontal plane and about the vertical axis), while in other embodiments, the acquired panoramic images or other images may include less than 360° vertical coverage (e.g., for images having a width that exceeds the height exceeding typical aspect ratios, such as at or exceeding 21:9 or 16:9 or 3:2 or 7:5 or 4:3 or 5:4 or 1:1, including so-called 'ultra-wide' lenses and resulting ultra-wide images). In addition, it should be understood that a user viewing such a panoramic image (or other image with sufficient horizontal and / or vertical coverage so that only a portion of the image is displayed at any given time) may be allowed to move the viewing direction within the panoramic image to different orientations so that different subset images (or "views") are presented within the panoramic image, and in some cases, such a panoramic image may be presented in a spherical coordinate system (including if the panoramic image is represented in a spherical coordinate system and a particular view is being presented, the image being presented is converted to a planar coordinate system, such as a perspective image view before being displayed). In addition, acquisition metadata regarding the capture of such a panoramic image may be obtained and used in various ways, such as data obtained from an IMU sensor or other sensor of a mobile device when the user carries the mobile device or moves the mobile device between acquisition locations. Non-exclusive examples of such acquisition metadata may include one or more of the following: acquisition time; acquisition location, such as GPS coordinates or other location indication; acquisition direction and / or orientation; relative or absolute acquisition order for multiple images acquired for a building or otherwise associated, etc., and such acquisition metadata may further optionally be used as part of determining the acquisition location of the image in at least some embodiments and situations, as further discussed below.The following includes additional details regarding the automated operation of one or more devices implementing an Image / Data Capture and Analysis (IDCA) system involved in acquiring images and optionally acquiring metadata, including with respect to. Figure 1 and 2A to Figure 2B and elsewhere in this article.

[0024] As described above, in various embodiments, the shapes of the rooms of a building may be automatically determined in various ways. For example, in at least some embodiments, a Mapping Information Generation Manager (MIGM) system may analyze various images acquired in and around a building to automatically determine the room shapes (e.g., 3D room shapes, 2D room shapes, etc.) of the rooms of the building and automatically generate a floor plan of the building. As an example, if multiple images are acquired within a particular room, the images may be analyzed to determine the 3D shape of the room in the building (e.g., reflecting the geometry of surrounding structural elements of the building), which analysis may include, for example, automated operations to “register” the camera positions of the images in a common reference frame, thereby “aligning” the images, and estimating the 3D positions and shapes of objects in the room, for example, by determining features visible in the content of such images (e.g., determining the direction and / or orientation of the capture device when the particular image was taken, the path of the room traveled by the capture device, etc., such as by using SLAM techniques for multiple video frame images and / or other SFM techniques for a “dense” set of images that are separated by at most a defined distance (e.g., 6 feet) to generate a 3D point cloud for the room including 3D points along the walls of the room and at least some of the ceiling and floor of the room, and optionally with 3D points corresponding to other objects in the room, etc.); and / or by determining and aggregating information about planes and normal (orthogonal) directions to those planes for detected features to identify objects in the room. The invention also provides a method for determining the estimated room shapes of the rooms in the building, wherein the plurality of room shapes are located together to form a floor plan and / or other related mapping information for the building, such as by connecting the various room shapes, optionally based at least in part on information about doorways and stairs and other inter-room wall openings identified in particular rooms, and optionally based at least in part on information about the determined path of travel of the mobile computing device between rooms. Similar techniques may be used to determine inter-location pose information for images captured at multiple locations, as discussed in more detail elsewhere herein. The following includes additional details regarding the automated operation of an apparatus implementing a MIGM system that relates to determining room shapes and combining room shapes to generate a floor plan, including information about the determined room shapes and the path of travel of the mobile computing device between rooms. Figure 1 and Figure 2D and elsewhere in this article.

[0025] For illustrative purposes, some embodiments are described below in which particular types of information are targeted to particular types of structures and are acquired, used, and / or presented in particular ways using particular types of devices, however, it will be understood that the described techniques may be used in other ways in other embodiments and thus the invention is not limited to the exemplary details provided. As a non-exclusive example, although in some examples a floor plan of a house may be generated that does not include detailed measurements for a particular room or for the entire house, it should be understood that other types of floor plans or other mapping information may be similarly generated in other embodiments, including for buildings (or other structures or layouts) that are separate from the house. As another non-exclusive example, although a floor plan for a house or other building may be used for display to help a viewer navigate the building, the generated mapping information may be used in other ways in other embodiments. As yet another non-exclusive example, while some embodiments discuss acquiring and using additional data from a mobile computing device that is separate from the camera device that captured the target image, in other embodiments, one or more devices used in addition to the camera device may have other forms, such as using a mobile device that acquires some or all of the additional data but does not provide its own computing capabilities (e.g., an additional "non-computing" mobile device), multiple separate mobile devices (whether mobile computing devices and / or non-computing mobile devices) that each acquire some additional data, etc. In addition, the term "building" refers herein to any partially or completely enclosed structure that typically but not necessarily includes one or more rooms that visually or otherwise separate the interior space of the structure, non-limiting examples of such buildings include a house, an apartment building or individual apartments therein, a dormitory, an office building, a commercial building, or other wholesale and retail structures (e.g., a shopping mall, a department store, a warehouse, etc.). As used herein, the terms "acquire" or "capture" with respect to a building interior, acquisition location, or other location (unless the context clearly indicates otherwise) may refer to the recording, storage, or recording of any media, sensor data, and / or other information related to spatial and / or visual characteristics and / or other perceptible characteristics of a building interior or a subset thereof, such as by a recording device or by another device that receives information from the recording device. As used herein, the term "panoramic image" may refer to a visual representation based on or separable into multiple discrete component images that originate from substantially similar physical locations in different directions and describe a larger field of view than any discrete component image described individually, including images with sufficiently wide-angle viewing angles from the physical location to include angles exceeding the angles perceptible from a person's gaze in a single direction (e.g., greater than 120° or 150° or 180°, etc.).As used herein, the term "sequence" of acquisition locations generally refers to two or more acquisition locations, each of which is visited at least once in a corresponding order, whether or not other non-acquisition locations are visited between them, and whether or not the access to the acquisition locations occurs during a single continuous time period, or occurs at multiple different times, or is performed by a single user and / or device, or by multiple different users and / or devices. In addition, various details are provided in the drawings and text for exemplary purposes, but are not intended to limit the scope of the invention. For example, the sizes and relative positions of the elements in the drawings are not necessarily drawn to scale, with some details omitted and / or greater prominence provided (e.g., by size and positioning) to enhance readability and / or clarity. In addition, the same reference numerals may be used in the drawings to identify similar elements or actions.

[0026] Figure 1 Included are example block diagrams of various computing devices and systems that may participate in the described techniques in some embodiments, such as an illustrated example of a portion of a building 198 on real estate 183 (in this example, house 198 on real estate 183), and a Building Floor Plan Generator and Location Determiner and Presenter (BFPGLDP) system 140 that is executed at least in part on one or more server computing systems 180 in this example embodiment.

[0027] In the illustrated embodiment, the IDCA system 150 obtains a target image 155 captured by one or more camera devices 184 at each of one or more image acquisition locations in each of one or more buildings, such as a 360° panoramic image captured using one or more camera devices that are designed to simultaneously capture 360° horizontal visual coverage, or otherwise use one or more lenses in a collection to capture 360° horizontal visual coverage. The IDCA system also obtains additional data 155 captured at each of one or more other data capture locations in each of one or more buildings by one or more mobile data capture devices 185, wherein the mobile data capture devices 185 move independently of the camera device 184 (e.g., not mounted to or otherwise physically connected together, such as to enable either device to be moved without moving the other device), the mobile data capture devices 185 shown in this example being mobile computing devices that include computing capabilities, the additional captured data including GPS location data (e.g., using one or more GPS receiver sensors 134 and / or other absolute location data, and may further include, for example, geographic orientation data (e.g., using a compass sensor 148c), device motion data (e.g., using one or more sensor modules 148, such as portions of an IMU sensor), additional images (e.g., using one or more imaging systems 135), and the like.

[0028] The BFPGLDP system 140 obtains images and other captured data 155 from the IDCA system 150 and uses it to determine absolute location data 156 for one or more locations in each of the buildings, although in other embodiments the system 140 may directly control the capture of some or all of such data. Whether in addition to or in lieu of the IDCA system. The BFPGLDP system 140 further uses the visual data of the captured images to determine room shapes of surrounding rooms, optionally combined with some additional captured data (e.g., device motion data for a mobile data capture device), and combines the determined room shapes to generate an associated building floor plan 165, such as by using corresponding functionality of the MIGM system 160, although in other embodiments the system 140 may directly control some or all of such generation of the building floor plan, whether in addition to or in lieu of the MIGM system. The BFPGLDP system 140 also automatically determines specific GPS location data or other absolute location data to associate with each generated floor plan 159, either during or after floor plan generation, including: in at least some embodiments, by automatically determining information 157 regarding image acquisition locations of one or more camera devices and data capture locations of one or more mobile data capture devices (e.g., based on analysis of individual data captured by the camera devices). Receive GPS data signals, etc. from the data capture devices, and extend absolute location data for at least some of the data capture locations to at least some of the image capture locations, such as by determining and using mapping data 158 between image capture locations in a local coordinate system for the building floor plan and other data capture absolute locations (e.g., pairs of one such image capture location and one such data capture absolute location, e.g., based at least in part on corresponding acquisition times, etc.) - whereby the information 157 and 158 can be used to extend absolute location data from the data capture locations of the mobile data capture devices to the image acquisition locations of the camera devices and the room shapes surrounding them.

[0029] In at least some embodiments and scenarios, automatic determination by the BFPGLDP system 140 (and by the IDCA system and / or MIGM system if the BFPGLDP system uses functionality of the IDCA system and / or MIGM system, respectively, for data capture and floor plan generation) is performed concurrently with data capture (e.g., in real-time or near real-time, such as within milliseconds, seconds, minutes, etc. of data capture), including: generating a partial building floor plan (e.g., incrementally expanding the floor plan by room shapes for each room in which an image and additional data is captured); and optionally using such partial building floor plans and / or other acquired and generated data to provide feedback to one or more operator users of one or more camera devices and / or one or more mobile data capture devices, including, in some embodiments and scenarios, displaying corresponding information in a GUI shown on a mobile data capture computing device. In some embodiments and scenarios, the BFPGLDP system 140 may optionally further use supporting information provided by system operator users via computing devices 105 over one or more intermediary computer networks 199.

[0030] In some embodiments, the IDCA system 150 and / or the MIGM system 160 may be executed on the same one or more server computing systems 180 as the BFPGLDP system (e.g., all systems are operated by a single entity or are otherwise executed in collaboration with one another, such as by integrating some or all functionality of all systems together), and in some embodiments, the IDCA system 150 and / or the MIGM system 160 may be operated on one or more other systems separate from the one or more systems 180 (e.g., on one or more mobile data capture devices 185 and / or other computing systems not shown), either instead of or in addition to copies of those systems executing on the one or more systems 180 (e.g., causing a copy of the MIGM system 160 executing on the device 185 to incrementally generate at least a partial building floor plan as building images are acquired by the IDCA system 160 executing on the device 185 and / or by that copy of the MIGM system, while another copy of the MIGM system is optionally executed on one or more server computing systems to generate a final complete building floor plan after all images are acquired, etc.). In the illustrated embodiment, a client application 154 for one or more of the BFPGLDP system and / or IDCA system and / or MIGM system may be executed on a capture device 185 (and in other embodiments and situations, some or all of the entire BFPGLDP system and / or IDCA system and / or MIGM system may be executed on some or all mobile devices 185, such as in a distributed manner), and a BFPGLDP client application or other building information viewer system (not shown) may be executed on one or more user client devices 175 to receive and present the generated building floor plan information and optionally associated information (e.g., a map of the surrounding area). In addition, in some embodiments, building information may be obtained by the BFPGLDP system in a manner other than via the IDCA and / or MIGM system (e.g., if such IDCA and / or MIGM system is not part of the BFPGLDP system), such as receiving building images and / or other data from other sources, and / or generating floor plans without using the MIGM system. The system 140 may also optionally store and use other data 143 including information about users of the capture devices 185 and / or camera devices 184 and / or other client devices 175 (e.g., as part of an associated account at the BFPGLDP system), such as preference-related data (e.g., used to personalize information and / or functionality provided to the user, including feedback related to data capture activities, presentation of generated information, etc.). Additional details related to the automatic operation of the BFPGLDP system are included elsewhere herein, respectively with respect to Figure 2C to Figure 2V and FIG. 4A to FIG. 4BAdditional details related to the automated operation of the IDCA and MIGM systems are included elsewhere herein, including with respect to Figure 5 and FIG. 6A to FIG. 6B .

[0031] The various components of the mobile data capture computing device 185 are also Figure 1, including: one or more hardware processors 132 (e.g., CPU, GPU, etc.), using executable instructions stored and / or loaded on one or more memory / storage components 152 of the device 185 to execute software (e.g., IDCA and / or MIGM and / or BFPGLDP application 154, optional browser or other software program, etc.); and one or more types of one or more imaging systems 135 (e.g., including one or more cameras with one or more lenses and one or more image sensors) for acquiring visual data of additional images (not shown, such as straight perspective images). In some embodiments, Device 185 may also receive some or all of target images 155 from one or more separately associated camera devices 184 (e.g., via a temporary wired / wired connection, Bluetooth or other device-to-device wireless communication, etc.), and provide storage and / or transmission functionality for these target images, whether in addition to or in lieu of images acquired by mobile device 185, and such as mobile device 185 optionally acting as a mobile companion device to the camera device and optionally having a wireless transmitter and / or receiver for exchanging wireless transmissions with a corresponding wireless receiver and / or transmitter on or associated with camera device 184. The illustrated embodiment of the mobile device 185 also includes one or more sensor modules 148, which in this example include a gyroscope 148a; an accelerometer 148b and a compass 148c (e.g., as part of one or more IMUs or inertial measurement units on the mobile device, not separately shown); one or more control systems 147 that manage I / O (input / output) and / or communications and / or networking of the device 185 (e.g., for receiving instructions from a customer and presenting information to the customer), such as to other device I / O and communication components 151 (e.g., a network interface or other connection, a keyboard, a mouse or other pointing device, a microphone, a speaker, a GPS receiver, etc.); a display system 149 (e.g., including one or more displays, optionally with a touch-sensitive screen); a GPS (or global positioning system) receiver / sensor 134 or other position-determining sensor (not shown in this example), optionally one or more depth sensing sensors or one or more types of other ranging components 136, optionally other components (e.g., one or more lighting components), etc. The other devices / systems 105, 175, and 180 and / or the camera device 184 may each include various hardware components and stored information in a manner similar to the mobile device 185, which are not shown in this example for the sake of brevity, and as described below with respect to Figure 3 Further, in other implementations and scenarios, at least some of the mobile data capture devices may lack some of the components shown, such as lacking computing power.

[0032] One or more users (e.g., end users, not shown) of one or more mobile client devices 175 may further interact with the BFPGLDP system 140 (and optionally the IDCA system 150 and / or the MIGM system 160), and / or with some or all of the BFPGLDP system executed on the device 175 (not shown), via one or more computer networks 199, to participate in using one or more cameras of the device 175 to acquire additional imagery within or around a building, or otherwise provide user-provided information, present or otherwise display received building data, and the like. Such mobile devices 175 may each execute a BFPGLDP client application or other building information viewer system (not shown) for interacting with the BFPGLDP system to request and receive building information, present such received building information and / or other received information on the mobile device (e.g., as part of a GUI displayed on the mobile device), and further optionally receive and respond to one or more user interactions with the presented information (e.g., using displayed user-selectable controls, such as portions of generated visual data enhancements), as discussed in more detail elsewhere herein, including with respect to FIG. 7A to FIG. 7BDiscussion of. Interactions with one or more users may include, for example: displaying a map with one or more 2D or 3D building floor plan models superimposed at locations corresponding to absolute locations associated therewith, specifying criteria for providing building information (e.g., criteria regarding building properties of interest to the user); obtaining and optionally requesting information for one or more indicated buildings (e.g., the building at which the user's mobile device is located, such as by providing one or more additional images obtained at the building); and interacting with the corresponding provided building information; Non-exclusive examples of interactions with displayed or otherwise presented information include the following: viewing building information, such as portions of provided descriptive building data; Selecting user-selectable controls provided with the provided building data may include, for example, interacting with one or more displayed visual indicators and / or textual descriptions associated with a particular building object or other building attribute in a visual data enhancement overlaid on a target image, such as obtaining further data related to the building object or other building attribute; changing between views and a plan view of a particular image at an acquisition location within or near the plan view; changing the horizontal and / or vertical viewing direction of a corresponding view of a displayed panoramic image to determine the portion of the panoramic image to which the current user viewing direction is directed, zooming and / or otherwise manipulating a displayed map and / or a building plan view model superimposed on a map, etc.). Additionally, the plan view (or portions thereof) may be linked to or associated with one or more other types of information, including, for a plan view of a multi-story or multi-level building, multiple associated sub-plan views of different floors or levels for interconnecting (e.g., by connecting stairways), a two-dimensional ("2D") plan view to be linked to a three-dimensional ("3D") or otherwise associated with a three-dimensional ("3D") rendering of a building, etc. Additionally, although in Figure 1 Not shown, but in some embodiments, client device 175 (or other devices, not shown) may receive and use information about a building (e.g., an identified floor plan and / or other mapping-related information) in an additional manner to control or assist in automated navigation activities by such devices (e.g., by an autonomous vehicle or other device), rather than or in addition to displaying the identified information.

[0033] exist Figure 1In the computing environment shown, network 199 may be one or more publicly accessible linked networks, possibly operated by various different parties, such as the Internet. In other implementations, network 199 may have other forms. For example, network 199 may instead be a private network, such as a company or university network that is completely or partially inaccessible to non-privileged users. In other implementations, network 199 may include private networks and public networks, wherein one or more private networks may access one or more public networks and / or may be accessed by one or more public networks. In addition, network 199 may include various types of wired and / or wireless networks in various circumstances. In addition, client device 175 and server computing system 180 may include various hardware components and stored information, as described below with reference to Figure 3 for a more detailed discussion.

[0034] As described above, the IDCA system may perform automated operations involving generating multiple 360° panoramic images at multiple associated image acquisition locations (e.g., in multiple rooms or other locations within a building or other structure, and optionally around some or all of the exterior of the building or other structure), such as using visual data acquired via one or more camera devices 184, and for generating and providing a representation of the interior of the building or other structure. For example, in at least some such embodiments, such techniques may include: using one or more such camera devices (e.g., a camera having one or more fisheye lenses and / or other lenses and mounted on a rotatable tripod or with an automatic rotation mechanism; a camera having sufficient fisheye lenses and / or other lenses to capture 360° horizontally without rotation; a camera of a smartphone or separate device held by or mounted on a user or the user's clothing and using one or more non-fisheye lenses, such as a wide-angle rectilinear lens and / or a telephoto lens and / or a macro lens and / or a standard lens, etc.), acquiring data from a sequence of multiple acquisition locations within multiple rooms of a house (or other building); and optionally further acquiring data involving movement of the capture device (e.g., movement at the acquisition locations, such as rotation; movement between some or all of the acquisition locations, e.g., for linking multiple acquisition locations together, etc.), in at least some cases, without measuring distances between the acquisition locations or having other measured depth information to objects in the environment surrounding the acquisition locations (e.g., without using any depth sensing sensors). After obtaining information about the acquisition location, the technology may include generating a 360° panoramic image from the acquisition location, the panoramic image having 360° horizontal information around a vertical axis (e.g., a 360° panoramic image showing the surrounding rooms in an equirectangular format); and then providing the panoramic image for subsequent use by the MIGM and / or BFPGLDP system.

[0035] In addition, the floor plan (or portion thereof) may be linked to or otherwise associated with one or more additional types of information, such as one or more associated and linked images or other associated and linked information, including a separate 2D model floor plan of a building that is linked to or otherwise associated with a separate 2.5D model floor plan rendering of the building and / or a 3D model floor plan rendering of the building, etc., and including a floor plan of a multi-story or other multi-level building having multiple associated sub-floor floor plans for different floors or levels that are interconnected (e.g., via connecting stairways), or are portions of a common 2.5D and / or 3D model. Thus, non-exclusive examples of end-user interaction with a displayed or otherwise generated 2D floor plan of a building may include one or more of the following: changing between the floor plan and a view of a particular image at a capture location within or near the floor plan; changing between a 2D floor plan view and a 2.5D or 3D model view, the 2.5D or 3D model view optionally including images of walls of the displayed model texture-mapped; changing the horizontal and / or vertical viewing direction of a corresponding subset view of a displayed panoramic image (or a portal into the panoramic image) to determine the portion of the panoramic image in a 3D coordinate system to which the current user viewing direction points; and rendering a corresponding floor plan image showing the portion of the panoramic image that does not have curvature or other deformations present in the original panoramic image, etc. Additionally, although in Figure 1 Not shown, but in some embodiments, client device 175 (or other devices, not shown) may additionally receive and use the generated floor plan and / or other generated mapping-related information to control or assist in automated navigation activities of those devices (e.g., by an autonomous vehicle or other device), either in lieu of displaying the generated information, or in addition to displaying the generated information.

[0036] Figure 1 Also shown is an exemplary building environment in which data may be acquired for use in the techniques, such as by moving through a building and acquiring 360° panoramic images of a target and / or other target images, and by moving through a building and acquiring additional data for determining GPS location data or other absolute location data, other data capture devices 185, which may be captured by the IDCA system for use by the MIGM system (e.g., under the control of the BFPGLDP system) to generate and provide one or more corresponding building floor plans (e.g., multiple incremental partial building floor plans) and / or further use such building information as part of an automated building information generation operation. In particular, Figure 1One floor of a multi-story house (or other building) 198 is shown having an interior that is at least partially acquired via multiple target panoramic images, such as when one or more associated camera devices 184 move through the interior of the building, the camera device 184 moving to a sequence of multiple acquisition locations 210 (e.g., starting at acquisition location 210A, moving along a travel path 115 to acquisition location 210B, moving along a travel path to acquisition location 210C, etc., and ending at acquisition location 210-O or 210P outside the building), and having additional data (e.g., absolute location data, additional images, etc.) captured by one or more mobile data capture devices 185, which move separately through the building to multiple other data capture locations. Embodiments of the IDCA system may automatically perform or assist in acquiring data representing the interior of a building (and further analyzing the acquired data to generate a 360° target panoramic image to provide a visual representation of the interior of the building), and embodiments of the MIGM system may analyze the visual data of the acquired images and, optionally, other acquired data to generate one or more building floor plans (e.g., multiple incremental floor plans) for the house 198, such as based at least in part on the determined direction and, optionally, based on the distances between multiple image acquisition locations (e.g., based on the directions between acquisition location pairs 210A and 210B, 210A and 210C, and 210B and 210C, respectively, and optionally based on the distances 215-AB, 215-AC, and 215-BC between location pairs 210A and 210B, 210A and 210C, respectively). Although such a mobile data capture device may include various hardware components, such as, for example, a camera, one or more sensors (e.g., gyroscopes, accelerometers, compasses, etc., such as part of one or more IMUs of the mobile device, or inertial measurement units; altimeters; light detectors, etc.), a GPS receiver, one or more hardware processors, memory, a display, a microphone, etc., in at least some embodiments, the mobile device may not have access to or use a device to measure the depth of objects in the building relative to the location of the mobile device, such that in such embodiments, the relationship between additional images and their associated data capture locations may be determined in part or in whole based on elements in different additional images without using any data from any such depth sensor, while in other embodiments, such depth data may be used. In addition, although in Figure 1GPS coordinate data 108 (e.g., for a specified location on real estate and / or at a building, such as northwest corner 195-1 of building 198, the northwest corner of a building, etc., and shown in this example using decimal degree format) and a direction indicator 109 are shown for the reader's reference relative to the exemplary house 198, but in at least some embodiments, the camera device and / or IDCA system may not use such absolute direction information, for example, in such embodiments, actual geographic direction is not taken into account to determine relative directions and distances between acquisition locations 210, while in other embodiments, such absolute direction information may be obtained and used.

[0037] In operation, camera device 184 arrives at a first acquisition location 210A within a first room within a building (in this example, in a living room accessible via exterior door 190-1) and acquires a target image having a view of a portion of the building interior visible from acquisition location 210A (e.g., some or all of the first room, and optionally small portions of one or more other adjacent or nearby rooms. For example, through a doorway wall opening, a non-doorway wall opening, a hallway, a stairway, or other connecting passageway from the first room). Similarly, mobile device 185 arrives at one or more different data capture locations in the first room, at which mobile device 185 acquires additional data, such as GPS location data and optionally additional images, as shown in reference to FIG. Figure 2C The image acquisition may be performed in various ways as discussed herein, and may include visual data regarding a plurality of structural elements or other objects visible from the acquisition location. Figure 1 In the example of , such objects within the building 198 include: walls; floors; ceilings; doorways 190 (including 190-1 to 190-6, such as having swinging doors and / or sliding doors); windows 196 (including 196-1 to 196-8); boundaries between walls and other walls / ceilings / floors, such as corners or edges 195 between walls (including corner 195-1 in the northwest corner of the building 198, corner 195-2 in the northeast corner of the first room, corner 195-3 in the southwest corner of the first room, corner 195-4 in the southeast corner of the first room, corner 195-5 at the north edge of the room passage between the first room and the corridor); furniture 191-193 (e.g., recliner 191; chair 192; table 193, etc.); pictures or paintings or televisions or other hanging objects 194 hung on the walls (e.g., 194-1 and 194-2); lamps ( Figure 1 various built-in appliances or other fixtures or other structural elements ( Figure 1), and the visual data may also show non-doorway wall openings 263a and 263b and 263c, etc. If the mobile device captures additional visual data in the living room, the additional visual data may similarly show some or all of such objects, but from one or more other data capture locations that are different from acquisition locations 210A and 210B and have different corresponding perspectives. The operator user may also optionally provide a text or auditory tag identifier to be associated with the acquisition location and / or surrounding rooms, and / or a descriptive annotation having one or more phrases or sentences about the room and / or one or more objects in the room, where the acquisition location and / or surrounding rooms are, for example, one of the acquisition locations 210A or 210B or the "living room" of the room including acquisition locations 210A and / or 210B, while in other embodiments, the IDCA and / or MIGM system may automatically generate such identifiers and / or annotations (e.g., by automatically analyzing images and / or videos of the building and / or other recorded information to perform corresponding automatic determinations, such as by using machine learning; based at least in part on input from an IDCA and / or MIGM system operator user, etc.) or may not use an identifier.

[0038] After the first acquisition location 210A has been acquired, the camera device 184 may be moved or moved under its own power to the next acquisition location (e.g., acquisition location 210B), and the mobile device 185 may be similarly moved or moved under its own power to the next data capture location, optionally recording images and / or video and / or other data from hardware components (e.g., from one or more IMUs, from cameras, from GPS sensors, etc.) during the movement between locations. At the next acquisition location, the one or more camera devices 184 may similarly acquire 360° target panoramic images and / or other types of target images from the acquisition location, and the one or more mobile devices 185 may similarly capture additional data from the one or more next data capture locations. The process may be repeated for some or all rooms of the building 198 and, in some cases, portions of the real estate 183 outside the building, as shown in this example for additional acquisition locations 210C-210P, including in this example: acquiring the target panoramic image and related additional other data at the following locations: on an exterior deck or patio or balcony area 186; on a larger exterior backyard or patio area 187a; in a separate side parking area 187b; near or in an exterior attached accessory structure area 189 (e.g., a garage, shed, accessory living unit, greenhouse, shed, driveway, etc.), which may have one or more rooms, and / or one or more doorways (e.g., a garage, shed, accessory living unit, greenhouse, shed, driveway, etc.); in a forecourt 187c outside the exterior doorway 190-1 (e.g., during a different acquisition period than that used to acquire some or all other target images, e.g., images of acquisition locations 210A through 210-O are acquired in a substantially continuous manner in a single image acquisition that occurs over a period of time such as 5 minutes or 15 minutes or 30 minutes), and in other embodiments and scenarios, from other acquisition locations (not shown) on an adjacent street or roadway 181 and / or sidewalk 182, from one or more aerial locations (e.g., from drone 179, and / or aircraft, satellites, etc., not shown), etc. The acquired images at each acquisition location may also be further analyzed, including in some embodiments presenting or otherwise placing each panoramic image in an equirectangular format, either at the time of image acquisition or later, and further analyzed by the MIGM and / or BFPGLDP system in the manner described herein.

[0039] As the mobile device moves through the building, the mobile device may receive GPS signals 178 at some or all of the data capture locations and associate corresponding GPS location data with each such data capture location, although in some embodiments and scenarios, the mobile device may not be able to receive GPS signals at some of the data capture locations and may instead perform other actions to determine absolute location data for such data capture locations. For example, in some embodiments and scenarios, as part of determining the absolute location data, the mobile device may also interact with one or more devices 235 in the home to receive wireless transmissions from a device 235a with an associated absolute location (e.g., a Bluetooth beacon, a Wi-Fi transmitter, etc.), thereby identifying the device or other visual information 235b with an associated absolute location in the visual data of the second image of the mobile device (e.g., a marking on a wall), etc., and further use such location data as part of determining the absolute location data for one or more data capture locations of the mobile device, whether in addition to or instead of using the GPS location data. However, in at least some such embodiments, the camera device does not have a GPS receiver and does not receive any GPS signals, or alternatively receives some GPS signals but does not have sufficient data to determine its own GPS location with sufficient accuracy (e.g., below a defined distance or uncertainty threshold). In at least some embodiments, one or more additional devices may also be present in the building (e.g., a drone device 179 inside or outside the building) that also receives GPS signals 178 and optionally acquires additional visual data, and if so, the additional visual data and associated GPS location data (or other absolute location data) captured by such one or more additional devices at additional locations may be used in combination with the one or more camera devices and the one or more mobile devices to determine the room shape, as well as to extend the absolute location data to the image acquisition location and / or to the floor plan (e.g., to the room shape within the floor plan) for further use of triangulation as part of determining the absolute location data for the image acquisition location in the manner described herein.

[0040] refer to Figure 1 Various details are provided, but it is to be understood that the details provided are non-exclusive examples included for purposes of illustration and that other implementations may be performed in other ways without some or all of such details.

[0041] Figures 2A to 2VAn example of automatically generating a building floor plan is shown, where the automatic generation is based at least in part on analyzing visual data of target images captured by a camera device at multiple acquisition locations in a building to determine room shapes of rooms surrounding the acquisition locations, and based at least in part on determining GPS location data or other absolute locations and associating the GPS location data or other absolute locations with the generated floor plan based on additional data captured at other locations in the building by an independent mobile device that moves independently of the camera device (e.g., by extending the absolute location data from the data capture location to other nearby image acquisition locations of the camera device and to surrounding room shapes visible in the acquired images), and for otherwise presenting the generated building floor plan overlaid on a map.

[0042] In particular, Figure 2A An exemplary target image 250a is shown, for example, as captured by camera device 184 from Figure 1 A non-panoramic perspective image (or a north-facing subset view formatted in a rectilinear manner of a 360-degree panoramic image 255a taken from the image acquisition location) taken in a northeast direction of an image acquisition location 210B in a living room of a house 198 of FIG. 1 , in which a direction indicator 109a is further displayed to show the northeast direction in which the image was taken. In the illustrated example, the displayed image includes several visual elements (e.g., a light fixture 130a), furniture (e.g., a chair 192-1), two windows 196-1, and a picture 194-1 hanging on the north wall of the living room. No inter-room passages into or out of the living room (e.g., doorways or other wall openings) are visible in the image. However, multiple room boundaries are visible in the image 250a, including a horizontal boundary between a visible portion of the north wall of the living room and the ceiling and floor of the living room, a horizontal boundary between a visible portion of the east wall of the living room and the ceiling and floor of the living room, and an inter-wall vertical boundary 195-2 between the north wall and the east wall.

[0043] Figure 2A Also shown is an exemplary target image 255a, such as a 360° panoramic image 255a acquired by camera device 184 from image acquisition location 210B, the target image 255a showing the entire living room in an equirectangular format. Since the panoramic image does not have the same orientation as perspective image 250a, the direction indicator 109 is not displayed for image 255a, although the pose information of the panoramic image may include one or more related directions (e.g., the start and / or end directions of the panoramic image, such as if acquired by rotation). The portion of the visual data of panoramic image 255a corresponds to the first perspective image 250a (shown generally in the center portion of image 255a), while the left portion of image 255a and the far right portion of image 255a include data corresponding to the perspective image 250a. Figure 2B 250b and 250c shown in the example, so that, for example, starting with image 255a, various perspective images including some or all of images 250a through 250c (and optionally a large number of other images) may be presented. This exemplary panoramic image 255a includes windows 196-1, 196-2, and 196-3, furniture 191-193, doorways 190-1 and 190-6, and a non-doorway wall opening 263a leading to a hallway room (the opening showing a portion of doorway 190-3 visible in an adjacent hallway). Image 255a also shows various room boundaries in a manner similar to perspective image 250a, but the horizontal boundaries are displayed in an increasingly curved manner, becoming more curved the farther they are from the horizontal centerline of the image, and the visible boundaries include vertical wall boundaries 195-1 to 195-4, a vertical boundary 195-5 located on the left / north side of the corridor opening, a vertical boundary on the south / right side of the corridor opening, and horizontal boundaries between the walls and the floor and between the walls and the ceiling.

[0044] Figure 2B Continue to show Figure 2A , and shows a camera device 184 in the northwest direction from Figure 1 A second perspective image 250b captured at acquisition location 210B in the living room of house 198 (wherein direction indicator 109b is further displayed to show the northwest direction from which the image was taken), and a second perspective image 250b captured by camera device 184 in the southwest direction at Figure 1 198 (wherein direction indicator 109c is further displayed to show the southwest direction from which the image was taken). As previously described, images 250b and 250c may alternatively be presented from respective subset portions of panoramic image 255a. In example image 250b, a small portion of one of windows 196-1 continues to be visible, along with a portion of window 196-2 and new light fixture 130b, and the horizontal and vertical room boundaries are similar to Figure 2A In example image 250c, portions of window 196-2 continue to be visible, as do recliner 191 and visible horizontal and vertical room boundaries, and further wall opening access to / from the living room is shown (in this example, this access is doorway 190-1 into and out of the living room, in Figure 1 It should be understood that various other perspective images may be captured from acquisition location 210B and / or other acquisition locations and displayed in a similar manner.

[0045] Figure 2C Shows Figure 1Other information 255c of the portion of the house 198, including the living room and a limited portion of other rooms to the east of the living room. Figure 1 and FIG. 2A to FIG. 2B As discussed, in some embodiments, target panoramic images may be captured at different locations within the house, such as at locations 210A and 210B in the living room along the path 115 traveled by the camera device 184, and the corresponding visual data of one or more such resulting target panoramic images may be subsequently used to determine the room shape of the living room. As further discussed with reference to subsequent figures. In addition, in at least some embodiments, absolute location data (e.g., GPS location data) and optional additional data (e.g., compass geographic direction data, IMU-based motion data, additional images, etc.) may be captured by the mobile data capture device 185 at one or more data capture locations along the path 116, in which example the mobile device 185 travels along different paths 116 in the living room, including a sequence of possible data capture locations 216 along the path where one or more additional images and absolute location data may be captured, wherein some specific such data capture locations 216a-216c are indicated, although in other embodiments and situations, the path 116 may be the same or substantially the same as the path 115. In some embodiments, as the mobile device 185 moves along the path, at least some of the additional images may be captured, such as if the mobile device is capturing a video having a sequence of video frame images (e.g., at 30 or 60 frames per second) or other image sequences (e.g., continuous or near continuous images) and corresponding absolute location data (e.g., GPS location data). In this example, locations 216 along the path 116 are shown as being separated by short distances (e.g., feet, inches, fractions of inches, etc.), although it will be appreciated that the video capture may be substantially continuous and thus, in at least some embodiments, only a subset of such captured video frame images (or other images from a continuous or near continuous image sequence) may be selected and used for further analysis, such as images separated by a defined distance and / or separated by a defined amount of time between their capture (e.g., seconds, fractions of a second, multiple seconds, etc.) and / or separated based on other criteria.

[0046] Figure 2CAlso shown is information 201 having an example of how absolute location data from a selected data capture location 216 may be extended to a target image captured at image capture location 210 - in this example, image capture location 210B has been associated with data capture location 216a (e.g., based on having the same or similar capture time), such as if data capture location 216a was selected from a set of data capture locations captured during a time window around the target image capture time. Image acquisition location 210B (e.g., from a set of data capture locations including at least 216a-216c). In this example, there is a distance 238c between locations 210B and 216a such that a simple assignment of GPS data for location 216a (in this example, 47.56292DD, -122.11705DD) without further refinement may result in an error of multiple feet in the actual absolute location of image acquisition location 210B (in this example, 47.56294DD, -122.11705DD), as shown in FIG. Figure 2F ). However, by determining a rigid transformation of multiple image acquisition locations on the building, the BFPGLDP system can eliminate or significantly reduce such translation errors, and further refinement activities can further improve accuracy, as discussed in more detail elsewhere herein. This extension of the absolute location data from the data capture location 216a to the image acquisition location 210B can be further extended to some or all of the surrounding room shapes visible in the target images acquired at the image acquisition locations 210A and 210B, for example to determine the absolute location of the northwest corner 195-1 of the living room (and building 198), in this example at 47.56296DD, -122.11805DD, as shown. Figure 2M ). Such room shape can be determined at least in part based on visual data of target panoramic image 255a captured at image acquisition location 210B and additional visual data of target panoramic image captured at image acquisition location 210A, as further discussed with respect to subsequent figures. For example, exemplary lines of sight 228 are shown from location 210B to various exemplary features in the room, and similar exemplary lines of sight (not shown) from location 210A to the same or other features can be determined and used in determination of room shape. Visual data of one or more such target panoramic images acquired by the camera device can be further analyzed as part of determining an estimated room shape for the room, such as based on identifying planes corresponding to walls and wall structural elements (e.g., windows, doorways, boundaries between walls, etc.), as discussed in more detail elsewhere herein.

[0047] Figure 2D continue FIG. 2A to FIG. 2C, and additional information 255d (including information 255d1-255d5) is shown as an example of analyzing multiple target images captured by the camera device along path 115 for determining the room shape of the surrounding room. In particular, visual data from multiple target images captured in the room can be combined as part of determining the estimated room shape of the room, optionally combined with IMU data captured simultaneously, for example, analysis of multiple images from locations along path 115 can be performed using SLAM and / or MVS and / or SFM techniques to provide various information about features of the living room, including information about 3D point clouds (e.g., such as Figure 2E ), and further analysis can be used to determine the relevant surface planes and normal orthogonal directions of the features from these planes.

[0048] In particular, information 255d1 shows features of the northeast portion of the living room visible in multiple target images, and information 255d2 further shows similar information about features in the northwest portion of the living room visible in the same or other target images, with various example features shown (e.g., corners 195-1 and 195-2, windows 196-1 and 196-2, etc.). As part of automated analysis of the target images (e.g., using SLAM and / or MVS and / or SFM techniques), information about planes 286e and 286f corresponding to portions of the north wall of the living room may be determined based on the detected features, and information 287e and 285f about portions of the east and west walls of the living room may similarly be determined based on corresponding features identified in the images. In addition to identifying such surface plane information for detected features (e.g., for each point in the sparse 3D point cloud determined from the image analysis), the technique may also determine information about possible acquisition poses (positions and orientations / directions) 220 and 222 of one or more target images (e.g., pose position 220g and optionally direction 220e in information 255d1 for acquisition location 210x1, and corresponding pose position 220g and optionally direction 220f in information 255d2, and corresponding pose position 220g and optionally direction 222f in information 255d2 for acquisition location 210x2d). Although only features of a portion of the living room are shown in information 255d1 and 255d2, it should be understood that other portions of the image corresponding to other portions of the living room may be analyzed in a similar manner to determine possible information about possible planes of various walls of the room and other features (not shown) in the living room. Additionally, a similar analysis may be performed between some or all other images captured in the living room, resulting in various determined feature planes from the various image analyses that may correspond to walls of the room.

[0049] Information 255d3 further illustrates information regarding various determined characteristic planes that may correspond to the west and north walls of the living room, derived from analysis of images captured from at least positions 210x1 and 210x2, and optionally further from images captured at other locations (e.g., images captured by a camera device from image location 210B). The plane information illustrated includes a determined plane 286G near or at the north wall (and a corresponding possible location of the north wall), and a determined plane 285G near or at the west wall (and a corresponding possible location of the west wall). In this example, there are many variations in the different determined planes for the north and west walls, depending on different features detected in the image analysis, such as differences in position, angle, and / or length, resulting in uncertainty regarding the actual precise position and angle of each wall. Although not illustrated in information 255d3, it will be appreciated that similar determined characteristic planes for other walls of the living room may be similarly detected, along with determined characteristic planes corresponding to features that are not along the walls (e.g., furniture). Information 255d4 further illustrates additional determined characteristic planes that may correspond to the west and north walls of the living room, by analyzing various other images captured by the camera device at image acquisition locations along the path 115 in the living room and / or captured by the mobile device at data capture locations along the path 116 in the living room, the analysis of which additional images in this example providing even greater variation in different determined planes for the north and west walls. Information 255d4 further illustrates additional determined information for aggregating information about the various determined characteristic planes in order to identify possible locations 295a and 295b of the west and north walls, as shown in information 255d5. In particular, information 255d4 includes indications 291a of normal orthogonal directions of some of the determined characteristic planes corresponding to the west wall, along with additional information 290a about those determined characteristic planes. In an exemplary embodiment, the determined feature planes are clustered to represent assumed wall locations for the west wall, and information about the assumed wall locations is combined to determine possible wall locations 295a, for example by weighting information from the various clusters and / or the feature planes determined below. In at least some embodiments, the assumed wall locations and / or normal information are analyzed using machine learning techniques to determine resulting possible wall locations, optionally by further applying assumptions or other constraints (such as 90° corners, as shown at 282 in information 255d3, and / or having flat walls) as part of the machine learning analysis or analysis results. A similar analysis can be performed for the north wall using information 290b about the corresponding determined feature planes and additional information 291b about the resulting normal orthogonal directions for at least some of those determined feature planes.The final possible wall positions 295a and 295b for the west wall and the north wall of the living room are shown in the information 255d5, respectively.

[0050] Despite Figure 2D Although not shown in FIG. 1 , a similar analysis may be performed for the other walls and optionally the floor and / or ceiling of the living room, and similarly for each of the other rooms of the building, thereby providing an estimated room shape for each of the rooms. Figure 2D Although not shown in the figure, in some embodiments, the analysis of visual data captured by the mobile computing device may be supplemented and / or replaced by analysis of depth data captured by the mobile computing device in the living room (not shown) to generate an estimated 3D point cloud directly from the depth data representing the walls and optionally the ceiling and / or floor of the living room, and / or to generate an estimated planar surface directly from the depth data representing the walls and optionally the ceiling and / or floor of the living room. Figure 2D Also not shown, but in at least some embodiments, other room shape estimation operations may be performed using only a single target panoramic image, such as by analyzing visual data of the target panoramic image through one or more trained neural networks, as discussed in more detail elsewhere herein.

[0051] Figure 2E Continue to show FIG. 2A to FIG. 2D 255e about additional information that can be generated from one or more images of a room and used in one or more ways in at least some embodiments. In particular, images (e.g., video frames) captured in a living room of house 198 can be analyzed to determine an estimated 3D shape of the living room, such as from a 3D point cloud of features detected in the video frames (e.g., using SLAM and / or SFM and / or MVS techniques, and optionally further based on IMU data captured by mobile device 185). In this example, information 255e is presented in a format similar to Figure 2B250c, reflecting an example portion of such a point cloud for a living room, such as corresponding to the northwest portion of the living room in this example (e.g., including the northwest corner 195-1 of the living room, and at least a portion of the window 196-1). Such a point cloud may be further analyzed to determine plane regions, such as corresponding to walls, ceilings, floors, etc., and in some cases to detect features, such as windows, doorways, and other room openings, etc., in this example, a first plane region 298 corresponding to the north wall of the living room is identified, and a second plane region 299 corresponding to at least a portion of the window 196-1 is further identified. It should be understood that in other embodiments, such an estimated 3D shape of the living room may be determined by using depth data captured by the mobile device 185 in the living room, whether additionally or alternatively, visual data of one or more images captured by the camera device 184 and / or the mobile computing device 185 in the living room may be used. In addition, it should be understood that various other walls and other features in the living room and other rooms of the house 198 may be similarly identified.

[0052] Figure 2F Continue to show FIG. 2A to FIG. 2E The example shown, and further information 230f is shown, the information 230f shows Figure 1 198, and an additional second image 255f acquired by the mobile device 185 at the data capture location 216b. In this example, the second image 255f is acquired from the location 216b at the orientation direction 243b, such as the target image 255a (at the acquisition location 210B) acquired by the camera device. Figure 2A184 ), and the second image 255f is a 180° panoramic image (shown in a rectilinear perspective format), although in other cases the second image 255f may be a non-panoramic linear image (e.g., example image 250d, shown as a subset of image 255f in this example). As shown in the example additional image 255f, the visual data of the additional image may include some or all of the visible portion of the camera device 184, in which example, the automatic determination may include: identifying the offset direction and distance 238F between the location 240b where the image 255f was acquired and the location 210B where the target image 255a was acquired, such as 2 feet east and 4 feet north from the data capture location 240b to the image acquisition location 210B. In this example, the mobile device also captures GPS coordinates 47.56292, -122.11698 (using decimal format) for the data capture location 216b, as shown in information 230f, and these absolute location data coordinates are subsequently expanded to the determined precise GPS coordinates 47.56294, -122.11697 for the image acquisition location 210B, which reflects the offset distance 238F therebetween, as discussed further below.

[0053] Figure 2G Continue to show FIG. 2A to FIG. 2FThe example shown in FIG. 2 shows one or more types of estimated room shape information 237 of the living room, based on an analysis of at least visual data of one or more target images captured by a camera device in the living room by the BFPGLDP system, and optionally further based on the visual data of one or more second images captured by the mobile device in the living room, wherein the one or more target images are, for example, target panoramic images 255a captured from the image acquisition location 210B. In particular, in this example, the estimated room shape information 237 includes information 237a about structural wall elements identified in the visual data of the one or more target images (e.g., wireframe lines of the room outline), and / or information 237b for displaying a 2D estimated room shape of the living room by analyzing the visual data of the one or more target images, and / or information 237c for displaying a 3D estimated room shape of the living room by analyzing the visual data of the one or more target images. In this example, the locations of structural wall elements such as windows, doorways, and other wall openings may be automatically identified and shown on the room shape information 237b and / or 237c, and additionally, the locations of one or more image acquisition locations of one or more target images used in the room shape determination may be further determined within the room shape, as shown for image acquisition location 210B, such as by directions and distances 236a and / or 236b between the image acquisition locations and the locations of walls and / or other portions of the determined one or more room shapes. As discussed in greater detail elsewhere herein, the estimation of the room shape of the living room based on the visual data of the target images may be performed in a variety of ways.

[0054] Figure 2G Further shown is one or more additional types of estimated room shape information 239 for the living room based on the analysis of additional data captured by the mobile device by the BFPGLDP system. In particular, in this example, the estimated room shape information 239 includes information 239a about structural wall elements identified from the analysis of the visual data of the one or more second images, and / or information 239b showing a 2D estimated room shape of the living room by analyzing the visual data of the one or more second images, and / or information 239c showing a 3D estimated room shape of the living room by analyzing the visual data of the one or more second images. In this example, the estimated room shape information 239a and 239b include solid lines to correspond to the information in the image. Figure 2F255f of the present invention, and the estimated room shape information 239c only includes information corresponding to that portion of the living room, however, if other additional images (not shown) have visual data for other portions of the living room, the combination of visual data from multiple additional images may further include information shown in dashed lines in the estimated room shape information 239a and 239b, and may be used to add additional information to the estimated room shape information 239c. In this example, the location of structural wall elements such as windows, doorways, and other wall openings may be automatically identified and shown on the room shape information 239b and / or 239c, and in addition, the location of one or more data capture locations of one or more additional second images used in the room shape determination may be further determined within the room shape, as shown for the data capture location 216b. It should be understood that the automatic determination of the estimated room shape information 237 and 239 may not produce a perfect room shape, so that there may be differences between the estimated room shape information 237 and 239. Additionally, although not shown with respect to information 239 , the locations of one or more data capture locations at which one or more second images used in the room shape determination were acquired may be further determined within the room shape.

[0055] Figure 2H Continue to show Figures 2A to 2G The example shown includes information 255h showing that when comparing Figure 2GThe plurality of estimated room shapes 237 and 239 are some further automatic operations of the BFPGLDP system to combine the analysis of one or more target images captured by the camera device (e.g., target panoramic image 255a from image acquisition location 210B, additional target panoramic images from image acquisition location 210A, etc., and optionally from analysis of one or more second images captured by the mobile device (e.g., from data capture location 216b). As discussed in more detail elsewhere herein, various matching criteria may be used to determine a fit between two room shapes. As an example, the lower right corner 247 may be initially matched, the estimated room shape may be measured and the determined The distance between one or more (e.g., all) matching corners of the room shape, such as shown at 234b for the two corners in the upper left (northwest) corner, where smaller distances reflect better matches (and, in some embodiments, higher matching scores), although not shown for other corners in this example, their distances for matching corners can be similarly measured and evaluated. As another example, the distance between one or more (e.g., all) corresponding wireframe structure lines of the estimated room shape and the determined room shape can be measured, such as shown at 234c for the two wireframes corresponding to the boundary of the north wall to the floor, with smaller distances reflecting better matches (and, in some embodiments, higher matching scores). In some embodiments, the distance between corresponding lines may be similarly measured and evaluated, although for other structural lines in this example not shown, the distance between corresponding lines may be similarly measured and evaluated. As yet another example, the distance between one or more (e.g., all) structural wall elements identified in the estimated room shape may be measured (e.g., using corresponding 3D bounding boxes for those structural wall element objects), such as 234a shown for the two doorways on the lower left side of the room, and smaller distances may reflect better matches (and, in some embodiments, higher match scores), although in other embodiments the entire bounding box area may be compared (e.g., using a joint upper intersection distance measurement), although in this example , but their distances between corresponding bounding box portions may be similarly measured and evaluated. As another example, the distances between one or more (e.g., all) corresponding walls of multiple room shapes may be measured, such as 234d shown for two walls on the top side of the room shape, where smaller distances reflect better matches (and, in some embodiments, higher match scores), although other walls are not shown in this example, but their distances between corresponding walls may be similarly measured and evaluated. Various other criteria may be accessed and used, either in addition to or in place of some or all of the criteria shown, as discussed in more detail elsewhere herein.Various distance offsets between multiple room shapes may be combined and used to determine the location 241a of the target panoramic image and optionally determine the acquisition orientation 243a within the room shape information, although it should be understood that for each of the acquisition locations and orientations. Figure 2H Only two degrees of freedom are shown in FIG. 1 , but the actual determined acquisition location and orientation information can be determined with three degrees of freedom for each of the acquisition location and orientation (e.g., the height of the acquisition location above the floor, and the angle upward or downward from the acquisition location height of the acquisition location). In addition, once the acquisition location information is determined for such a target image, it can be displayed on an updated version of the floor plan of the building, such as Figure 2N The information 255N is shown in the updated plan views 230n and 265n.

[0056] Fig.2I Additional information 255i is shown, corresponding to, after estimated room shapes have been determined for the rooms on the illustrated floor of the house 198, positioning the estimated room shapes 229a-229h of the rooms relative to each other based at least in part on connecting room pathways between the rooms and matching room shape information between adjacent rooms, and in at least some embodiments, such information may be processed as constraints on room positioning and optimal or otherwise preferred solutions determined for those constraints. Fig.2I Examples of such constraints in include: matching 231 the connection channel information of adjacent rooms (e.g., in FIG. 2C to FIG. 2E The present invention also provides a method of matching 232 adjacent room shapes so as to connect the shapes (e.g., as shown for rooms 229d and 229e and for rooms 229a and 229b) to the corridors detected in the automatic image analysis discussed above so that the locations of these corridors are co-located; and matching 232 adjacent room shapes so as to connect the shapes (e.g., as shown for rooms 229d and 229e and for rooms 229a and 229b). In addition to or as an alternative to corridor-based constraints and / or room shape-based constraints, various other types of information may be used in other embodiments, such as exact or approximate dimensions for the overall dimensions of the house (e.g., based on available additional metadata about the building, analysis of images from one or more image acquisition locations outside the building, etc.). House exterior information 233 may be further identified and used as a constraint (e.g., based at least in part on automatic identification of corridors and other features (e.g., windows) corresponding to the exterior of the building) to prevent another room from being placed at a location that has been identified as the exterior of the building.

[0057] Figure 2J to Figure 2L Continue to show FIG. 2A to FIG. 2I The example shown here shows the FIG. 2A to FIG. 2I The mapping information generated by the analysis types discussed in . Specifically, Figure 2JAn exemplary floor plan model 230j is shown that may be constructed based on the location of a determined estimated room shape, which in this example includes indications of walls and doorways and windows. In some embodiments, such a floor plan may have further information shown, such as regarding other features that are automatically detected by the analysis operation and / or subsequently added by one or more users. For example, Figure 2KA modified floor plan 230k is shown, which includes various types of additional information, such as can be automatically identified based on analysis operations of visual data from images and / or depth data and added to floor plan 230j, including one or more of the following types of information: room labels (e.g., "Living Room" for a living room); room dimensions; visual indications of fixtures or appliances or other built-in features; visual indications of the location of additional types of associated and linked information (e.g., information about panoramic images and / or perspective images acquired at a specified acquisition location, which can be selected by the end user for further display; audio annotations and / or recordings in audio annotations and / or recordings selectable by the end user for further presentation, etc.); visual indications of doorways and windows, etc. In other embodiments and scenarios, some or all of this type of information may alternatively be provided by one or more MIGM system operator users and / or IDCA system operator users and / or BFPGLDP system operator users. Additionally, when displaying floor plans 230j and / or 230k to an end user, one or more user-selectable controls may be added to provide interactive functionality as part of a GUI (graphical user interface) screen such as screen 255k, such as indicating the current floor displayed, allowing the end user to select a different floor to be displayed, etc., wherein a corresponding example user-selectable control 228 is added in this example. Additionally, in some embodiments, changes to floors or other levels may also be made directly from the displayed floor plan, such as by selecting a corresponding connecting passage (e.g., a staircase to a different floor), and other visual changes may be made directly from the displayed floor plan by selecting a corresponding displayed user-selectable control (e.g., selecting a control corresponding to a specific image at a specific location, and receiving a display of that image instead of, or in addition to, a previous display of the floor plan in which the image was selected). In other embodiments, information for some or all of the different floors may be displayed simultaneously, such as by displaying separate sub-floors for separate floors, or by integrating room connection information for all rooms and floors into a single floor plan that is shown together. It should be understood that various other types of information may be added in some embodiments, that some of the types of information shown may not be provided in some embodiments, and that visual indications and user selections of linked and associated information may be displayed and selected in other manners in other embodiments.

[0058] Figure 2L Continue to show Figures 2A to 2K , and illustrates examples that can be generated and displayed (e.g., in a manner similar to Figure 2K230j and / or 230k, wherein additional information 265l is provided in the GUI of the GUI, which in this example is a 2.5D or 3D model floor plan of the house. Such a model 265l may be additional mapping-related information generated based on the floor plans 230j and / or 230k, wherein additional information about height is shown in order to show the visual location in the wall of features such as windows and doors. Although in Figure 2L Not shown, but for 2D floor plan models Figure 2K Some or all of the additional types of information shown in may similarly be provided in other formats such as Figure 2L The 3D floor plan model shown in FIG. Figure 2L Not shown, but in some embodiments, additional information may be added to the displayed walls, such as from images taken during video capture (e.g., by “texturing” the walls by presenting and showing actual paint, wall paper, or other surfaces from the house on the presented model 265), and / or specified colors, textures, or other visual information may be otherwise added to the walls and / or other surfaces.

[0059] Figure 2M Continue to show Figures 2A to 2L The example shown in FIG. 25 shows further information 255m about associating the absolute location data with the determined room shape and the corresponding generated floor plan. In particular, as previously described with respect to Figure 2C and Figure 2F As discussed, the mobile device determines GPS location data for multiple data capture locations, such as locations 216a and 216b, and can be extended to image acquisition locations, such as location 210B, such as shown in information 255m1. In addition, the location of such image acquisition location within the corresponding determined room shape can be determined, such as the distance and direction 238m1 to the northwest corner of the room (and the northwest corner of the building in this example), as shown in reference Figure 2G As discussed, in some embodiments and circumstances, similar distance and direction information may be similarly determined for data acquisition locations, such as if a second image is acquired from those data acquisition locations and used as part of the room shape determination and / or if depth data is captured from those data capture locations and used, such as the distance and direction 238m2 and 238m3 to the northwest corner of the room for data capture locations 216b and 216a, respectively. It should be understood that such extension of GPS location data or other absolute location data from an image acquisition location such as 210B to locations in the surrounding room shape may be performed for various room shape locations (e.g., each corner, each wall midpoint, each window or doorway, etc.) and may be associated with corresponding locations on the resulting generated floor plan in a variety of ways.

[0060] Figure 2MAlso shown is information regarding the use of GPS location data (or other absolute location data) associated with a generated floor plan to indicate the location of the floor plan on a larger map so that a visual representation of the associated absolute location data can be superimposed on the map (e.g., as a dot or other visual indicator of one or more specific floor plan points; by displaying the actual generated floor plan at its determined location; by displaying one or more building images, such as an above image, etc.). In the example information 255m2, a 2D floor plan of a building 198 is overlaid on or otherwise displayed as part of a larger map from a top-down perspective, and the example information 255m3 similarly displays a 3D model of the first floor of the building 198 on a map at a higher zoom level. Information 255m2 and 255m3 also illustrate how geographic direction information associated with the floor plan can be further used, including regarding a displayed geographic direction indicator 109m. It should be understood that such a map display may include various other types of information and functionality, including information regarding Figure 2Q-2V In other embodiments, such absolute location data and / or geographic direction data associated with a floor plan may be further used in various other ways.

[0061] Figure 2N Continue to show Figures 2A to 2M The example shown is further shown with information 255n regarding additional position and absolute location data for an additional image (not shown) acquired at a new acquisition location 216m in the living room of the house 198 after the building floor plan is generated. In this example, the visual data of the additional image can be analyzed in a manner similar to that discussed elsewhere herein to determine the relative position of the additional acquisition location, such as a relative position within the room shape of the living room and / or position-to-position gesture information between the additional acquisition location and a previous location in the room for determining location data (e.g., image acquisition location 210B, data capture location 216B, etc.). Previously determined GPS location data for the room (in this example, for the northwest corner of the room) and / or previous locations in the room for determining location data (in this example, one or both of locations 210B and 216B) can be used to extend the previously determined GPS location data to the new acquisition location 216m, and corresponding information can be displayed (e.g., overlaid on the 2D floor plan 230m and / or on the 3D model 265m) to resemble Figure 2N The method shown, or provided to one or more recipients in other ways.

[0062] Figure 2-O (here referred to as " Figure 2-O " to prevent confusion with the number 20) Figure 2A-2N, and further information 2550 is shown to provide an example of a smoothing operation for absolute position data points captured in the living room of the example house 198. In particular, information 230o1 shows the original GPS data points as the mobile device moves around the living room, while information 230o2 shows the modified GPS data point information after the smoothing operation is performed. Obviously, in addition to removing outlier data, the smoothed data better shows the path that reflects the movement of the mobile device when the GPS data points were captured. In addition, a subset 2420 of GPS data points is shown, which is selected as potentially associated with the image acquisition location 210A, such as acquired in a time window close to the acquisition of a target image acquired for the location 210A, for determining representative GPS data points associated with the target image. Other details about performing such smoothing operations are included herein.

[0063] Figure 2P continue Figures 2A to 2-OAn example of a method of using additional data for an example house 198 to improve the accuracy of GPS data points for the interior of the house and / or to help determine structural information of the house, such as an exterior boundary, exterior door, driveway or other walkway leading to the house, etc., such as based at least in part on additional GPS data points captured during a traversal of one or more exterior areas around the house (e.g., based on the property on which the house is located). Specifically, information 242p1 shows an example of using absolute position data (e.g., detected motion trajectory information) captured from a data capture device for an exterior doorway of a house, such as to determine the absolute position of the exterior doorway based on a transition from the exterior of the house to the interior of the house (e.g., by analyzing visual data being captured, input from an operator at the time of the transition, etc.) and selecting associated absolute position data at that time - in this example, a series of captured absolute position data points from a mobile capture device are shown to show a path approaching an exterior doorway and continuing after passing through the exterior doorway. In addition, information 242p2 shows an example of using absolute position data captured from a data capture device to detect linear external motion and align it with an external walkway or driveway, such as determining the absolute position of an external walkway or driveway. In this example, a series of captured absolute position data points are shown to show a path approaching an external doorway corresponding to a walkway to the front doorway of a house. Information 242p3 shows an example of using additional human operator activity data to improve GPS signal quality, such as causing a human operator to track a specified external pattern before entering a building to improve GPS signal calibration, which will improve subsequent in-building GPS data points. In this example, a series of captured absolute position data points are shown to show a path outside a house before entering the house. Information 242p4 shows an example of using additional human operator activity data to cause a human operator to track some or all of the external boundaries of a building so that the shape of the generated floor plan can be matched to that boundary - in this example, a series of captured absolute position data points are shown to show a path outside the house.

[0064] Figure 2Q-2V continue Figure 2A-2P Examples of, and further information for providing an example of automatic operation of the BFPGLDP system to automatically present a building floor plan with associated absolute position data using surrounding reality data for the associated absolute positions of the building floor plan, such as in a GUI that optionally contains various additional types of information and / or types of functionality. Specifically, Figure 2QInformation 255q is shown including a map showing a plurality of houses in a neighborhood using the overhead imagery, including house 198. Additionally, a portion of a 2D floor plan model for the house is overlaid on the visible imagery of house 198. In this example, the overlaid 2D floor plan model includes only a single floor of the house and only structural elements of the house including walls, doorways, windows, and stairs are shown. Figure 2Q Additional information is also shown overlaid on a map proximate to the visual representation of the building floor plan, and nearby points of interest are displayed with information related to the location of the building floor plan (e.g., directions to nearby parks and the nearest highway) as well as directional information 109Q and absolute location data for the northwest corner of house 198, although in other embodiments and scenarios, some or all of such additional information may not be displayed (e.g., based on user selections and / or preferences) or may instead be overlaid on or otherwise included in the visual representation of the building floor plan. Figure 2R Shown as Figure 2Q Information 255R is an enlarged version of the map and shows additional elements of the 2D plane model, in particular, at the zoom level shown, similar to Figure 2K The way shows built-in structures and some fixtures and appliances such as in bathrooms and kitchens. Figure 2R Additional information on the coverage map is also shown to Figure 2Q Nearby points of interest are similarly shown, such as directions to a nearby park, as well as directional information 109r and absolute location data for the northwest corner of house 198, although in other embodiments and scenarios, some or all of such additional information may not be shown (e.g., based on user selections and / or preferences). Figure 2S Shown as Figure 2Q A further zoomed-in version of the map of information 255s is provided and further elements of the 2D plan model are shown therein, in particular, at the zoomed-in level shown, in addition to showing the built-in structures and some fixtures and appliances, furniture and other built-in fixtures (e.g., track lighting) and labels, these fixtures and appliances, furniture and other built-in fixtures (e.g., track lighting) and labels are also shown. For example in the living room. Furthermore, Figure 2S Also shown is the addition of additional information about the surrounding area, so as to provide information about the view from the north window in the living room, and additional information about the map, so as to provide a map similar to Figure 2Q Nearby points of interest are shown in a manner such as directions to a nearby park, as well as directional information 109s and absolute location data for the northwest corner of house 198. Although in other embodiments and scenarios, some or all of such additional information may not be shown (e.g., based on user selections and / or preferences). Figure 2TInformation 255T is shown showing a further alternative for presenting floor plan information overlaid on an overhead image of a house, and in other embodiments and scenarios, similar floor plan information may be overlaid on a street-level image of the front of the house. In these examples, information 255t1 shows an example of overlaying a 3D floor plan model of the first story of a house on a top image of the house, while information 255t2 shows an example of overlaying 2D floor plan information for multiple stories on a top image of the house, emphasizing in this example the second story using heavier lines for walls and additional types of information displayed (e.g., room labels), and in this example using different colors for structural elements of different stories. Figure 2U An alternative view is shown in which information 255U about a floor plan of a house overlaid on a map is shown as part of a set of information displayed about the house, such as a floor plan of the house overlaid on a map is shown along with multiple types of information in the GUI, corresponding to the end-user's selection of control 202U. In this example, the information displayed includes an array of images captured for the house (e.g., photographs 1, etc.) and other types of building information, which in this example include a street view of the house, a 3D computer model of the house (which in this example indicates the acquisition location and direction 219a of the acquired image), a 2D floor map model of the house, and the like. It should be understood that other types of building information can be similarly displayed, such as interactive tours, videos, animations, non-visual data (e.g., audio recordings), and the like. In this example, each image or other type of building information shown in the array can be selected by the user, for example, and for the selected type of building information can become the primary (or only) type of information shown in the updated GUI display. It should be understood that various other types of information can be similarly displayed via Figure 2U Such an interface obtains, such as textual information (e.g., facts about the building; a larger map of the surrounding area; a requested price for the building available for sale or other type of possible acquisition (e.g., lease); an estimated value of the building for sale or other type of possible acquisition, such as based on automatic determination using comparisons with data on other similar buildings or otherwise, etc.).

[0065] Figure 2VAnother view is shown in which information 255V about a house plan model is displayed along with information about one or more features of the exterior of the house in order to provide a "site plan" visualization of the property on which the house is located. In this example, a visual representation of the floor plan (in this case, the first layer of the 3D model) is shown along with information about the property on which the house is located and adjacent roads, as well as information about the property including vegetation, visual representations of sidewalks and other paved or covered or finished surfaces (e.g., a patio on the south side of the house, a deck on the east side of the house, etc.), information about adjacent features (e.g., a green belt on the east side of the property), an attached garage and optional other structures (not shown), etc. In this example, various exterior features are shown using geometric shapes or other abstractions, but in other embodiments, they may be shown in other ways, such as using actual images of individual features and / or all of the property. It should be understood that various other types of exterior features may be shown, and additional types of information about a property or other site may be included in some embodiments and situations.

[0066] Figure 3 is a diagram showing the implementation of the BFPGLDP system 140 (e.g., in a manner similar to Figure 11 and 12. The block diagram of an embodiment of one or more computing systems 180 for implementing an IDCA system 150 and a MIGM system 160 in a manner (eg, a manner) and one or more server computing systems 380 that perform implementations of the IDCA system 150 and the MIGM system 160, wherein the one or more computing systems 180 and the BFPGLDP system, and / or computing systems 380 and / or the IDCA and MIGM systems may be implemented using a plurality of hardware components forming electronic circuits adapted and configured to perform at least some of the techniques described herein when in combined operation. The mobile data capture devices 185 may each execute one or more client applications and / or other programs 154, and one or more other computing systems and devices may be capable of optionally executing a BFPGLDP system client application and / or other building information viewer system 396 (such as each mobile device 175) and / or optional other programs 335 and 383 (such as one or more server computing systems 180 and 380, respectively, in this example). In the illustrated embodiment, each server computing system 180 includes one or more hardware central processing units ("CPUs") or other hardware processors 305, various input / output ("I / O") components 310, storage 320, and memory 330, wherein the illustrated I / O components include a display 311, a network connection 312, a computer-readable media drive 313, and other I / O devices 315 (e.g., a keyboard, a mouse or other pointing device, a microphone, a speaker, a GPS receiver, etc.). Each server computing system 380 may have similar components, although for simplicity, only one or more hardware processors 381, storage 385, memory 384, and I / O components 382 are shown in this example.

[0067] One or more server computing systems 180 and executing BFPGLDP system 140, and one or more server computing systems 380 and executing IDCA and MIGM systems 150 and 160, and data capture device 185 and executing software 154, and mobile device 175 and executing software 396 can communicate with each other, and in the illustrated embodiment can communicate with other computing systems and devices, such as via one or more networks 199 (e.g., the Internet, one or more cellular telephone networks, etc.), including interacting with optional other navigable devices 395, which receive and use floor plans and optional other generated information for navigation purposes (e.g., use by semi-autonomous or fully autonomous vehicles or other devices), and for the capture device 185 to communicate with building equipment not shown (e.g., using communications and / or sensor components to receive transmissions from transmitter devices and / or otherwise communicate with other building equipment, such as electronic lock boxes or locks, smart home devices, etc.). In this example embodiment, the mobile device 175 is shown as including one or more displays 392 on which building information from the BFPGLDP system is provided, and optionally including other components 394 (e.g., computing resources, I / O components, sensors, etc.). In other embodiments, some of the described functionality may be combined in fewer computing systems, such as combining some or all of the BFPGLDP system 140 with a building information viewer system 396 in a single system or device (e.g., the mobile device 175), combining the data capture functionality of the IDCA and MIGM systems 150 and 160 and one or more devices 185 in a single system or device, combining the BFPGLDP system 140 with one or both of the IDCA and MIGM systems 150 and 160 in a single system or device, combining the BFPGLDP system 140 and the IDCA and MIGM systems 150 and 160 and the data capture functionality of one or more devices 185 in a single system or device, etc.

[0068] In the illustrated embodiment, an embodiment of the BFPGLDP system 140 is executed in the memory 330 of one or more server computing systems 180 to perform at least some of the described techniques, such as by using one or more processors 305 to configure the one or more processors 305 and the computing system 180 to execute software instructions of the system 140 in a manner that implements the described techniques. The illustrated embodiment of the BFPGLDP system may include one or more components (not shown) to each perform a portion of the functionality of the BFPGLDP system, and the memory may further optionally execute one or more other programs 335, as a specific example, in at least some embodiments, a copy of the IDCA and / or MIGM system may be executed as one of the other programs 335, for example, instead of or in addition to the IDCA and / or MIGM systems 150 and 160 on one or more server computing systems 380, and / or a copy of the building information viewer system may be executed as one of the other programs 335 (for example, if the one or more computing systems 180 are the same as the mobile device 175). The BFPGLDP system 140 may also store and / or retrieve various types of data on the storage device 320 (e.g., in one or more databases or other data structures) during its operation, such as acquired images / data 155, building floor plans and determined room shapes and associated wall element information 165, acquired absolute location data 156, data regarding image acquisition locations and data capture locations 157 and 158 (including inter-location pose data), generated floor plans and / or other mapping information and associated absolute location data 159 (e.g., generated and saved 2.5D models and / or 3D models, for use with associated floor plans, and / or other mapping information). Figure 1 ), and / or various types of optional additional information 329 (e.g., various analytical information related to the presentation or other use of one or more building interiors or other environments).

[0069] In addition, embodiments of the IDCA system 150 and the MIGM system 160 are executed in the memory 385 of one or more server computing systems 380 in the illustrated embodiment to perform techniques related to generating panoramic images and floor plans of buildings, such as by using one or more processors 381 to configure one or more processors 381 and one or more computing systems 380 to execute software instructions of the systems 150 and / or 160 in a manner that automatically performs these techniques. The illustrated embodiments of the IDCA system and the MIGM system may include one or more components not shown to perform portions of the functions of the IDCA system and the MIGM system, respectively, and the memory may further optionally execute one or more other programs 383. The IDCA system 150 and / or the MIGM system 160 may also store and / or retrieve various types of data on a storage device 384 (e.g., in one or more databases or other data structures) during operation, such as video and / or image information 155 acquired for one or more buildings (e.g., 360° video or images for analysis to generate floor plans to be provided to a user of a client computing device 370 for display, etc.), floor plans and / or other generated mapping information 165, and optional other information 387 (e.g., for use with associated floor plans). Figure 1 Additional images and / or annotation information used with the associated plane Figure 1 and other information related to the rendering or other use of one or more building interiors or other environments, and the like), and optionally interacting with or using information from one or more I / O components 362, although in Figure 3 Not shown, the IDCA system and / or MIGM system may also store and use additional types of information, such as other types of building information to be analyzed and / or provided to the BFPGLDP system, about IDCA system and / or MIGM system operator users and / or end users, etc.

[0070] Some or all of the mobile device 175, mobile data capture device 185, optional other navigable devices 395, other client devices 105, and other computing systems (not shown) may similarly include some or all of the same types of components shown for the server computing system 180. As a non-limiting example, the mobile data capture devices 185 are each shown to include one or more hardware CPUs 132, memory 367, storage devices 365, one or more GPS receiver sensors 134, one or more imaging systems 135 (e.g., for acquiring video and / or images), optional IMU hardware sensors 148 (e.g., for acquiring associated device movement data, etc.), optional one or more depth sensors 136, and optional other components (not shown). In the illustrated example, zero or more client applications 154 (e.g., applications specific to the IDCA system and / or specific to the MIGM system and / or specific to the BFPGLDP system) and / or other programs 154 are executing in the memory 367 to participate in communications with the BFPGLDP system 140, the IDCA system 150, the MIGM system 160, and / or other computing systems. Although specific components are not shown for other navigable devices 395 or other computing devices / systems 105, it should be understood that the other navigable devices 395 or other computing devices / systems 105 may include similar and / or additional components.

[0071] It should also be understood that Figure 3 The computing systems / devices 180 and 185 and 380 and 175 and other systems and devices included in the are merely illustrative and are not intended to limit the scope of the invention. The system and / or device may instead each include multiple interactive computing systems or devices, and may be connected to other devices not specifically shown, including via Bluetooth communication or other direct communication, via one or more networks such as the Internet, via the Web, or via one or more dedicated networks (e.g., mobile communication networks, etc.). More generally, the device or other computing system may include any combination of hardware that can interact and perform the functions of the type described, optionally when programmed or otherwise configured with specific software instructions and / or data structures, including but not limited to desktop computers or other computers (e.g., tablet computers, etc.), database servers, network storage devices and other network devices, smart phones and other cellular phones, consumer electronic devices, wearable devices, digital music player devices, handheld gaming devices, PDAs, wireless phones, Internet appliances, and various other consumer products including appropriate communication capabilities. Additionally, in some embodiments, the functionality provided by the illustrated BFPGLDP system 140 may be distributed among various components, some of the described functionality of the BFPGLDP system 140 may not be provided, and / or other additional functionality may be provided.

[0072] It should also be understood that, although various items are shown as being stored in memory or in storage devices when in use, these items or portions thereof may be transferred between memory and other storage devices for the purpose of memory management and data integrity. Alternatively, in other embodiments, some or all of the software components and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Thus, in some embodiments, when configured by one or more software programs (e.g., by the BFPGLDP system 140 executed on the server computing system 180, by the BFPGLDP client application or other building information viewer system executed on the mobile device 175 or other computing system / device, etc.) and / or data structures, some or all of the described techniques may be performed by a hardware device including one or more processors and / or memories and / or storage devices, such as by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures, and, for example, executing the algorithms described in the flowcharts and other disclosures herein. Additionally, in some embodiments, some or all of the systems and / or components may be implemented or provided in other ways, such as by including one or more devices implemented in part or in whole in firmware and / or hardware (e.g., rather than devices implemented in whole or in part by software instructions that configure a particular CPU or other processor), including but not limited to one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), a network storage device, or a portable media article (e.g., a DVD disk, a CD disk, an optical disk, a flash memory device, etc.) to be read by an appropriate drive or via an appropriate connection. In some embodiments, the systems, components, and data structures may also be transmitted via generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless-based and wire / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Therefore, embodiments of the present disclosure may be implemented with other computer system configurations.

[0073] FIG. 4A to FIG. 4BAn exemplary embodiment of a flow chart for a building floor plan generation and location determination manager (BFPGLDP) system routine according to an embodiment of the present disclosure is shown. The routine may be executed by, for example, Figure 1 and / or Figure 3 BFPGLDP system 140, and / or as described in Figure 2A-2V and the BFPGLDP system described elsewhere herein to perform automated operations associated with automatically generating a building floor plan and determining associated absolute locations of the generated floor plan using visual data and additional data from images captured within a building interior and, optionally, a building exterior by a plurality of data capture devices. Automatically associating absolute position data points from a data capture location of a mobile device with an acquisition location of the image so as to extend the absolute position data from the image acquisition location of the camera device to a surrounding room shape determined from the visual data of the image and, optionally, to additional floor plan locations (e.g., exterior corners of a building), and optionally using the floor plan's associated absolute position data to overlay a visual representation of the floor plan on an associated map. FIG. 4A to FIG. 4B In the exemplary embodiment shown, the indicated building may be a house or other type of building, and various types of information may be provided or otherwise used in a particular manner, but in other embodiments, other types of buildings and information uses may be provided and used, as discussed elsewhere herein.

[0074] The illustrated implementation of the routine begins at block 405, where instructions or information are received. The routine continues to block 407, where it is determined whether the instructions or other information received in block 405 indicate that absolute location data is to be associated with one or more devices at a building and / or other objects (e.g., objects within the building, such as beacons or other wireless transmitters, visual markers, etc., objects outside the building, and objects visible within the building, etc.), and if so, continues to block 409, where absolute location data for such devices and / or objects is determined or otherwise obtained and stored for later use. Such absolute location data may be automatically determined in a variety of ways, such as based on a public data source (e.g., for objects outside the building), data captured during installation and / or placement of devices and / or objects (e.g., for objects inside the building, such as visual markers), based on a mesh network of a plurality of such devices having associated transmitter and receiver capabilities and absolute location data of at least one such device, etc.

[0075] After box 409, or if it is determined in box 407 that the instructions or other information received in box 405 do not associate absolute location data with the device and / or object, the routine continues to box 410, where it is determined whether the instructions or information received in box 405 is to associate absolute location data with a floor plan and / or other building information for use with the indicated building, optionally after generating the floor plan and / or other building information. If not, continue to box 490. Otherwise, the routine continues to box 415 to retrieve one or more images with visual data and optionally acquired metadata for the building (e.g., orientation data and other posture data) and other absolute location data for the building (e.g., GPS data points for data capture locations), and otherwise execute an IDCA system routine to acquire images and optionally acquire metadata and other absolute location data for the building, and receive results from the routine - an example of such an IDCA reference Figure 5 Further discussion of the system routine. After block 415, the routine continues to block 420 to retrieve at least a portion of the floor plan information and additional information about the building and its properties, and otherwise executes a MIGM system routine to determine the floor plan and additional information about the building and its properties (e.g., based at least in part on the data obtained by the IDCA system routine), and receives results from the routine, an example of such a MIGM system routine will be referred to. FIG. 6A to FIG. 6B Further discussion.

[0076] After box 420, the routine continues to box 425, where it optionally performs an initial smoothing operation (e.g., to remove outliers) on the absolute position data points. In box 430, the routine then determines whether to determine additional absolute position data for one or more data capture locations using visual data and / or other data acquired at those data capture locations that are not absolute position data points, such as data capture locations for which GPS position data was not previously captured or otherwise determined, all data capture locations for which visual data and / or other data was acquired, etc. If so, the routine continues to box 435 to analyze, for each such data capture location, the visual data of any second image acquired from the data capture location to attempt to identify one or more visible objects with associated absolute locations and / or to identify other locations with known absolute position data using other captured data (e.g., transmissions received from in-building devices with known absolute locations), and determine corresponding absolute position data, objects, or other locations for the data capture location by extending the known absolute position data from the identified locations. In embodiments where a data capture location has associated GPS location data captured at the data capture location, in some embodiments such additional location data determined in box 435 may be used to supplement the captured GPS location data, such as where a GPS signal received by the mobile device at the data capture location is sufficiently weak to introduce uncertainty in the captured GPS location data.

[0077] After box 435, or if it is determined in box 430 that other absolute location data for any data capture location is not determined (e.g., using the GPS location data or other absolute location data retrieved from box 415), the routine continues to box 440 to determine, for each acquired image, a representative GPS data point or other absolute location data point to be paired with the image and its acquisition location (e.g., using the location of the acquisition location in the local coordinate system for the floor plan), such as selecting the best matching data point (e.g., based on the same or similar acquisition time, such as the absolute location data point with the most recent acquisition time), selecting multiple data points (e.g., having acquisition times in a time window around the acquisition time of the image), and generating a representative data point from them, etc. In box 445, the routine then determines a global transformation for data pairs in the first and second data sets, where the first data set includes image locations in the local coordinate system of the floor plan and the second data set includes associated absolute location data points, such as a best fit transformation that maps locations in the local coordinate system of the floor plan to the associated absolute location data points, and optionally using additional information (e.g., if a compass heading associated with the image is available for orienting the acquired image, such as using an algorithm that solves the Wahba problem and / or using a point set registration technique and / or using a trained machine learning model, and optionally including performing refinement using iterative closest point or another refinement technique. In box 475, the routine then uses the resulting global transformation to determine a global transformation for some or all of the floor plans (e.g., Absolute location data points for one or more floor plan points representing one or more exterior walls of a building) are determined, for example, by determining a revised absolute location data point for the image acquisition location and extending the absolute location data from the image acquisition location to locations in a corresponding room shape in the floor plan (e.g., one or more floor plan points representing one or more exterior walls of a building). Locations in a room shape using the image acquisition location) and other information (if any) having an associated location on the floor plan (e.g., other media captured at the particular capture location), including optionally determining absolute location data for one or more points on an exterior boundary (e.g., a corner) of the building, and optionally including geographic direction information (e.g., at least a north heading) relative to the floor plan.

[0078] After block 475, the routine continues to block 480 to determine whether to use the absolute location data associated with one or more points of the floor plan to associate the floor plan with the map, such as based on the instructions or other information received in block 405, and if so, to continue to block 485 to perform such map association, including overlaying or otherwise including a visual representation of the floor plan at its associated absolute location on the map. Optionally, fitting the visual representation to an existing visual representation of the building on the map (e.g., as part of an image that is part of the map or otherwise overlaid on the map), and optionally associating additional relevant information about the building (e.g., neighborhood information, directions to points of interest, etc. - as discussed in more detail elsewhere herein, various types of floor plan information may be presented or otherwise made visible based on various factors (e.g., based on the zoom level of the map, the size of the visual representation of the building floor plan, user selections and / or preferences, etc.). Following box 485, or if instead it is determined in box 480 that such map information association is not to be performed at the current time, the routine continues to box 489 where the routine optionally provides some or all of the determined and / or generated information for the routine to one or more recipients (e.g., providing overlay information of box 485 to a map for display on one or more client devices).

[0079] If it is determined in block 410 that the instructions or other information received in block 405 will not generate building information, the routine continues to block 490 to appropriately perform one or more other indicated operations. Such other indicated operations may include, for example, one or more of the following non-exclusive examples: receiving and storing (or otherwise determining) information about the known absolute location of a particular device and / or object; receiving and storing information about a building and / or capture device and / or companion device and / or user for later use; retrieving information about a user device and / or associated user from a BFPGLDP system account and providing it to the device or user, etc.

[0080] After frame 489 or frame 490, routine continues to frame 495, to determine whether to continue, such as until receiving the definite instruction to terminate, or only when receiving the definite instruction to continue. If it is determined to continue, routine returns to frame 405 to wait for additional instructions or other information, otherwise continues to frame 499 and ends.

[0081] Although not targeted FIG. 4A to FIG. 4BThe automatic operations shown in the exemplary embodiments of the BFPGLDP system are described, but in some embodiments, a human user may further help facilitate some of the operations of the BFPGLDP system, such as providing one or more types of input to an operator user and / or end user of the BFPGLDP system, which is further used for subsequent automatic operations. As non-exclusive examples, such a human user may provide one or more types of input, non-exclusive examples include the following: to provide input related to the location of equipment and / or objects installed or placed at a building location, to help identify objects and / or other attributes based on analysis of images, floor plans and / or other building information, etc., to help associate absolute data location data with image acquisition locations and / or room shapes and / or floor plans, etc. Additional details regarding embodiments in which one or more human users provide inputs used in additional automatic operations of the BFPGLDP system are included elsewhere in this document.

[0082] Figure 5 An example flow chart of an implementation of an IDCA (image capture and analysis) system routine 500 is shown. The routine may be implemented by, for example, Figure 1 and Figure 3 IDCA system 150, and / or as described in FIG. 2A to FIG. 2B and the IDCA system described elsewhere herein to acquire 360° panoramic images and / or other images at image acquisition locations within a building or other structure using one or more camera devices, and / or to acquire other data (e.g., GPS location data, other additional images, etc.) at other data capture locations within a building or other structure using one or more other mobile devices for use in subsequently generating related floor plans and / or other mapping information. Although portions of the example routine 500 are discussed with respect to acquiring specific types of images and other data at specific locations, it will be understood that this routine or similar routines may be used to acquire video (with video frame images) and / or other data (e.g., audio) instead of or in addition to such panoramic images or other perspective images and other data. Additionally, while the illustrated embodiment captures and uses information from the interior of a target building, it should be appreciated that other embodiments may perform similar techniques in other contexts, including for non-building structures and / or for information on the exterior of one or more target buildings of interest (e.g., on the property on which the target building is located, to display yards, decks, patios, outbuildings, etc.). Additionally, some or all of the routines may be performed on a mobile device used by a user to capture the image information, and / or by a system remote from such a mobile device. In at least some embodiments, the image information may be captured from a mobile device. FIG. 4A to FIG. 4B Box 415 of routine 400 calls routine 500, provides corresponding information from routine 500 to routine 400 as part of the implementation of box 415, and in this case, returns processing control to routine 400 at box 599 and / or after box 577 or 590. In other embodiments, routine 400 can perform additional operations in an asynchronous manner without waiting for such processing control to be returned (for example, performing other processing activities while waiting for corresponding information from routine 500 to be provided to routine 400).

[0083] The illustrated implementation of the routine begins at block 505, where instructions or information are received. At block 510, the routine determines whether the received instructions or information indicate the performance of directed acquisition of visual data and / or other data representing the interior of a building (e.g., based on information provided about one or more acquisition locations and / or other guided acquisition instructions), and if not, proceeds to block 590 to perform one or more other indicated operations, including, in some embodiments and situations, receiving one or more target images captured by one or more camera devices without directed acquisition and / or other data captured by one or more camera devices without directed acquisition. Otherwise, the routine proceeds to block 511 to optionally provide instructions or other information to one or more operator users involved in performing image acquisition and / or capturing absolute position data points, such as information to improve the capture of GPS data points or other absolute position data points (e.g., performing initial movement activities before entering a building to improve GPS calibration, collecting GPS data points at specific locations, such as entering a doorway and / or some / all of the building's exterior boundaries and / or. an exterior walkway or other exterior area, etc.). In box 512, the routine then receives an instruction to begin an image acquisition process by a camera device at a first image acquisition location in or around a building (e.g., from a human operator user of the camera device that will perform the target image acquisition, whether for the same or different user operating the camera device), and / or to begin capturing other data by a mobile device at a first data capture location (e.g., from a user of a mobile data acquisition device that will perform the data acquisition process). After block 512, the routine proceeds to block 515 to perform an image acquisition activity to acquire a 360° panoramic image of the image acquisition location at the target building of interest using the camera device (e.g., via one or more fisheye lenses and / or non-fisheye rectilinear lenses on the mobile device, and providing at least 360° horizontal coverage around the vertical axis, although in other embodiments, other types of images and / or other types of data may be acquired), and / or perform a data capture activity to acquire other data at the data capture location by the mobile device (e.g., capture GPS location data and optionally one or more additional second images, and optionally obtain IMU data and / or other capture metadata during the image capture activity) to simultaneously capture data by two devices at locations close to each other (e.g., within each other's visual range or otherwise with overlapping visual data). As a non-exclusive example, the camera device may be a rotating (scanning) panoramic camera equipped with a fisheye lens (e.g., with 180° horizontal coverage) and / or other lenses (e.g., with less than 180° horizontal coverage, such as a regular lens or a wide-angle lens or an ultra-wide lens or a macro lens).The routine may also optionally obtain annotations and / or other information from one or more users of the camera device and / or mobile device regarding corresponding image acquisition locations and / or data capture locations and, optionally, surrounding circumstances for later use in the presentation of information regarding one or more locations and / or surrounding circumstances.

[0084] After box 515 is completed, the routine continues to box 520 to determine whether there are more image acquisition locations and / or more data capture locations, at which target images are acquired using a camera device and at which other data are acquired using a mobile device, such as based on corresponding information provided by one or more users of one or more devices and / or received in box 505. In some embodiments, the IDCA routine will acquire only one or more target images captured by a camera device at a single image acquisition location and / or other data captured at a single data capture location, and then proceed to box 577 to provide those target images and / or other data and optional corresponding information (for example, to the BFPGLDP system and / or MIGM system for further use before receiving additional instructions or information to capture one or more next images at one or more next image acquisition locations and / or one or more other data groups at one or more next data capture locations). If there are more image acquisition locations to acquire additional images from the camera device at the current time and / or more data acquisition locations to acquire other data from the mobile device at the current time, the routine continues to box 522 to optionally initiate the acquisition of link information (e.g., acceleration data, visual data, etc.) during the movement of one or more devices from one or more current locations to one or more next locations of the building along one or more movement paths. The acquired link information may include additional sensor data (e.g., from one or more IMUs or inertial measurement units on the mobile device or otherwise carried by the user) and / or additional visual information recorded during such movement (e.g., images, videos, etc.). Initiating the acquisition of such link information may be performed in response to an explicit instruction from the device user or based on one or more automatic analyses of information recorded from the device. In addition, in some embodiments, during the movement to the next acquisition location, the routine may also optionally monitor the movement of the device and provide one or more guidance prompts (e.g., to the user) regarding the movement of the device, the quality of the sensor data and / or visual information being acquired, the relevant lighting / environmental conditions, the desirability of acquiring images and / or other data at the next location, and any other suitable aspects of acquiring link information. Similarly, the routine may optionally obtain annotations and / or other information about one or more travel paths from one or more users, such as for later use in presentation of information about the travel paths or resulting connections between locations.In box 524, the routine determines that the camera device has reached a next image acquisition location and / or the mobile device has reached a next data capture location (e.g., based on an indication from the user, based on the device ceasing forward movement for at least a predetermined amount of time, based on an amount of time that has elapsed since the last image acquisition and / or data capture, based on an amount of distance that has been moved since the last image acquisition and / or data capture, etc.), respectively, to use as the new current image acquisition location and / or data capture location, and returns to box 515 to perform further target image acquisition activities for the new current image acquisition location and / or further capture of other data for the new current data capture location.

[0085] If it is determined in box 520 that there are no other image acquisition locations at the current time to acquire additional target images of the current building or other structure, and there are no other data capture locations at the current time to acquire additional other data of the current building or other structure, the routine proceeds to box 545 to optionally pre-process the acquired 360° target panoramic image and / or other acquired data prior to subsequent use (e.g., to generate relevant mapping information for providing information about structural elements or other objects in a room or other enclosed area, etc.) to produce images of a specific type and / or specific format (e.g., performing an equirectangular projection for each such image, in which straight vertical data remains straight, such as the sides of a typical rectangular door frame or a typical boundary between two adjacent walls, and straight horizontal data, such as the top of a typical rectangular door frame or a boundary between walls, remains straight at the horizontal centerline of the image, but in the equirectangular projection image, gradually curves in a convex manner relative to the horizontal centerline as the distance from the horizontal centerline in the image increases and / or as the distance from the acquisition location decreases). At block 577, the image and other captured data and any related generated or obtained information are stored for later use and optionally provided to one or more recipients (e.g., to block 415 of routine 400 if called from this block). FIG. 6A to FIG. 6B One example of a routine for generating a floor plan representation of a building interior from captured images and other data is shown.

[0086] If it is determined in box 510 that the instruction or other information received in box 505 is not to use directional capture to acquire images and other data representing the interior of a building, the routine continues to box 590 to appropriately perform any other indicated operations to receive one or more target images captured by one or more camera devices at one or more image acquisition locations without directional capture, to receive other data captured by one or more other mobile devices at one or more data capture locations without directional capture, to respond to requests for generated and stored information (e.g., identifying one or more panoramic images that match one or more specified search criteria, etc.), to obtain and store other information about users of the system, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by users of the system, such as users of mobile devices that acquire the interiors of one or more buildings, operator users of an IDCA system, etc.), to perform any housekeeping tasks, etc.

[0087] After frame 577 or frame 590, routine proceeds to frame 595 to determine whether to continue, for example, until receiving explicit indication to terminate, or only when receiving explicit indication to continue. If it is determined to continue, routine returns to frame 505 to wait for additional instructions or information, and if not, proceeds to frame 599 and ends.

[0088] Although not targeted Figure 5 The automatic operation shown in the exemplary embodiment of the IDCA system is described, but in some embodiments, a human user may further help facilitate some of the operations of the IDCA system, such as an operator user and / or an end user of the IDCA system providing one or more types of input, which is further used in subsequent automatic operations. As a non-exclusive example, such a human user may provide one or more types of input as follows: provide input to help determine the acquisition location, such as providing input in box 512 and / or box 524, which is used as part of the automatic operation of the box; perform activities related to image acquisition in box 515 (e.g., participate in image acquisition, such as activating a shutter, implementing settings on a camera device and / or associated sensors or components, rotating a camera device, as part of acquiring a panoramic image, etc.; set the position and / or orientation of a camera device and / or associated sensors or components, etc.); perform activities related to other data capture in box 515 (e.g., participate in the capture of other data); provide input in box 515 and / or box 522, used as part of subsequent automatic operations, such as labels, annotations or other descriptive information about a specific image, surrounding rooms and / or objects in the room, etc. Additional details regarding implementations in which one or more human users provide input that is further used in other automated operations of the IDCA system are included herein.

[0089] FIG. 6A to FIG. 6B An exemplary embodiment of a flow chart of a MIGM (Mapping Information Generation Manager) system routine 600 is shown. The routine may be executed by, for example, Figure 1 and Figure 3 MIGM system 160, and / or as described in FIG. 2A to FIG. 2N and the MIGM system described elsewhere herein to determine a room shape of a room (or other defined area) by analyzing information from one or more images acquired in the room (e.g., one or more 360° target panoramic images, one or more additional second images, etc.) to generate a partial or complete floor plan of the building or other defined area based at least in part on the one or more images of the building or other defined area and optionally based on additional data acquired by the mobile computing device using the determined room shape, and / or to generate other mapping information of the building or other defined area based at least in part on the one or more images of the building or other defined area and optionally based on additional data acquired by the mobile computing device. FIG. 6A to FIG. 6B In the example of , the determined room shape of the room may be a 2D room shape representing the location of the walls of the room, or a 3D fully closed combination of planar surfaces representing the location of the walls and ceiling and floor of the room, or a 2.5D combination of planar surfaces representing the location of at least the walls of the room without complete ceiling and / or floor data, and the generated mapping information of the building (e.g., a house) may include a 2D floor plan and / or a 3D computer model floor plan and / or a 2.5D computer model floor plan, but in other embodiments, other types of room shapes and / or mapping information may be generated and used in other ways, including for other types of structures and defined areas, as discussed elsewhere herein. In at least some embodiments, the room shape may be generated from FIG. 4A to FIG. 4B Box 420 of routine 400 calls routine 600, wherein corresponding information from routine 600 is provided to routine 400 as part of the implementation of box 420, and processing control is returned to routine 400 at box 699 or, in this case, after boxes 688 and / or 699. In other embodiments, routine 400 can continue with additional operations in an asynchronous manner without waiting for such processing control to be returned (for example, once the corresponding information from routine 600 is provided to routine 400, wait to continue, continue other processing activities while waiting for the corresponding information from routine 600 to be provided to routine 400, etc.).

[0090] The illustrated implementation of the routine begins at block 605, where information or instructions are received. The routine continues to block 610 to determine whether image information and optionally other captured data is already available for analysis of one or more rooms (e.g., for some or all of the indicated building, such as based on one or more such images received in block 605 as previously generated by the IDCA routine), or whether such image information is currently to be acquired. If it is determined in block 610 that some or all of the image information is currently being acquired, the routine continues to block 612 to acquire such information, optionally waiting for one or more users or devices to move through one or more rooms of the building, and acquiring panoramic images or other target images at one or more image acquisition locations in the one or more rooms or other areas (e.g., multiple acquisition locations in each room of the building), and / or acquiring other second images and optionally other data at one or more data capture locations in the one or more rooms or other areas (e.g., multiple data capture locations in each room of the building), optionally together with metadata information about the acquisitions and / or interconnect link information related to movement between acquisition locations, as discussed in more detail elsewhere herein, implementation of block 612 may, for example, include calling an IDCA system routine to perform such activities, wherein Figure 5 An exemplary embodiment of an IDCA system routine for performing such image acquisition is provided. If it is determined in box 610 that images and optionally other data are not currently being acquired, the routine continues to box 615 to acquire one or more existing panoramic images or other target images from one or more image acquisition locations in one or more rooms or other areas (e.g., multiple images acquired at multiple acquisition locations including at least one image in each of the rooms of the building and the acquisition location) and to acquire existing other data captured at one or more data capture locations in one or more rooms or other areas, optionally together with metadata information about the acquisition and / or interconnect link information related to movement between acquisition locations, and optionally together with the determined location of the acquisition location, such as may be provided in box 605 together with corresponding instructions in some cases.

[0091] After either box 612 or 615, the routine continues to box 620 where it is determined whether mapping information for an interconnected collection of target panoramic images (or other images) for a building or other group of rooms is being generated (sometimes referred to as a "virtual tour" to enable an end user to move from any one of the images in the linked collection to one or more other images linked to the starting current image, including, in some embodiments, by selecting a user-selectable control for each such other linked image displayed with the current image, optionally by overlaying visual representations of such user-selectable controls and corresponding inter-image directions over the visual data of the current image, and similarly moving from a next image to one or more additional images linked to the next image, etc.), and if so, continues to box 625. The routine in box 625 selects at least some pairs of images (e.g., images based on pairs with overlapping visual content), and if acquisition location information has not been determined and provided, determines for each pair a relative orientation between the images of the pair based on shared visual content and / or other acquired link interconnection information (e.g., motion information) associated with the images of the pair (whether to move directly from the location where one image of the pair was acquired to the location where the other image of the pair was acquired, or instead to move between those starting and ending locations via one or more other intermediate locations of other images), which, if acquisition location location information has been determined and provided, can be used to determine relative orientation information between the image pairs, either in lieu of or in addition to visual data analysis. The routine in box 625 may further optionally use at least the relative orientation information of the image pairs to determine the global relative positions of some or all of the images in the images relative to each other in a common coordinate system, and / or generate inter-image links and corresponding user-selectable controls as described above. Additional details on creating such linked groups of images are included elsewhere herein.

[0092] Following box 625, or if it is determined in box 620 that the instruction or other information received in box 605 is not to determine a linked set of images, the routine continues to box 635 to determine whether the instruction received in box 605 indicates to generate other mapping information (e.g., a floor plan) for the indicated building, and if so, the routine continues to execute some or all of boxes 637-685 to do so, otherwise continuing to box 690. In box 637, the routine optionally obtains additional information about the building, such as from activities performed during the acquisition and optional analysis of the images, and / or from one or more external sources (e.g., an online database, information provided by one or more end users, etc.), such additional information may include, for example: exterior dimensions and / or shape of the building; additional imagery and / or annotation information acquired corresponding to specific locations on the exterior of the building (e.g., surrounding the building and / or for other structures on the same real estate, from one or more aerial locations, etc.); additional imagery and / or annotation information acquired corresponding to specific locations within the building (optionally, for locations different from the acquisition location of the panoramic image or other image); determined acquisition location information, etc.

[0093] After box 637, the routine continues to box 640 to select the next room (starting with the first) for which one or more images (e.g., 360° target panoramic image, other target images, other second images, etc.) acquired in the room are available, and analyze the visual data of the one or more images of the room to determine the room shape (e.g., by at least determining wall locations), optionally together with uncertain information about the walls and / or other parts of the room shape, and optionally including identifying other wall and floor and ceiling elements (e.g., wall structural elements / objects such as windows, doorways and stairs and other inter-room wall openings and connecting passages, wall boundaries between a wall and another wall and / or ceiling and / or floor, etc.) and their locations within the determined room shape, if acquisition location location information has been determined and provided, then this information can be used as part of determining the room shape information, either instead of or in addition to the visual data analysis. In some embodiments, room shape determination may include determining a 2D room shape using the boundaries of the walls to each other and at least one of the floor or ceiling (e.g., using one or more trained machine learning models), while in other embodiments, room shape determination may be performed in other ways (e.g., by generating a 3D point cloud of some or all of the room walls and optionally the ceiling and / or floor, for example, by analyzing at least visual data of a panoramic image and optionally additional data acquired by a mobile data capture device or optionally an associated mobile computing device, optionally using one or more of SfM (Structure from Motion) or SLAM (Simultaneous Localization and Mapping) or MVS (Multi-View Stereo) analysis. Additionally , the activities of box 645 may also optionally determine and use acquisition location information for each analyzed image (e.g., within the corresponding determined room shape), and / or obtain and use additional metadata for each panoramic image (e.g., acquisition height information relative to the floor and / or ceiling of the camera device or other mobile data capture device used to acquire the panoramic image). Additional details regarding determining room shapes and identifying additional information for rooms are included elsewhere herein. After box 640, the routine continues to box 645, where it is determined whether there are more rooms for which room shapes are determined based on the images acquired in those rooms, and if so, returns to box 640 to select the next such room for which to determine the room shape.

[0094] If it is determined in block 645 that there are no more rooms for which room shapes are generated, the routine continues to block 660 to determine whether to further generate at least a partial floor plan of the building (e.g., based at least in part on one or more determined room shapes from block 640, and if available, based on determined acquisition location location information, and optionally also based on information about how the determined room shapes are positioned relative to each other). If not, such as when only one or more room shapes are determined without generating other mapping information for the building (e.g., determining a room shape for a single room based on one or more images acquired by the IDCA system in the room), the routine continues to block 688. Otherwise, the routine continues to block 665 to retrieve one or more room shapes (e.g., the room shapes generated in block 645), or obtain one or more room shapes of the rooms of the building (e.g., based on manually provided input), whether 2D room shapes or 3D room shapes, and then continues to block 670. In block 670, the routine uses the one or more room shapes to create an initial floor plan (e.g., an initial 2D floor plan using 2D room shapes and / or an initial 3D floor plan using 3D room shapes), such as a partial floor plan that includes one or more room shapes but less than all room shapes of the building, or a complete floor plan that includes all room shapes of the building. If multiple room shapes are present, the routine in block 670 also determines the positioning of the room shapes relative to each other, such as by using visual overlap between images from multiple acquisition locations to determine the relative positions of those acquisition locations and room shapes surrounding those acquisition locations, and / or by using other types of information (e.g., using connecting room corridors between rooms, optionally applying one or more constraints or optimizations; utilizing determined acquisition location position information, etc.). In at least some embodiments, the routine in block 670 further refines some or all of the room shapes by generating a binary segmentation mask covering the relatively positioned one or more room shapes, extracting polygons representing the outlines or contours of the segmentation mask, and separating the polygons into the refined one or more room shapes. Such a floor plan may include, for example, relative position and shape information for various rooms without providing any actual size information for the individual rooms or the building as a whole, and may also include multiple linked or associated sub-maps of the building (e.g., to reflect different floors, levels, sections, etc.). The routine also optionally associates the locations of doors, wall openings, and other identified wall elements on the floor plan.

[0095] After box 670, the routine optionally performs one or more steps 680-685 to determine and associate additional information with the floor plan. In box 680, the routine optionally estimates the dimensions of some or all of the rooms, such as from analysis of the image and / or acquisition metadata of the image or from general dimensional information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan, it being understood that if sufficiently detailed dimensional information is available, an architectural drawing, blueprint, etc. can be generated from the floor plan. After box 680, the routine continues to box 683 to optionally associate further information with the floor plan (e.g., with a specific room or other location within the building), such as additional existing images with specified locations and / or annotation information. In box 685, if the room shape from box 645 is not a 3D room shape, the routine also optionally estimates the height of the walls in some or all of the rooms, such as based on analysis of the image and optionally based on the size of known objects in the image, and height information about the camera when the image was acquired, and uses the height information to generate a 3D room shape for the room. The routine further optionally uses the 3D room shapes (whether from box 640 or from box 685) to generate a 3D computer model floor plan of the building, where the 2D floor plan and the 3D floor plan are associated with each other, and in other embodiments, only the 3D computer model floor plan may be generated and used (including, if desired, by using horizontal slices of the 3D computer model floor plan to provide a visual representation of the 2D floor plan).

[0096] Following box 685, or if it is determined in box 660 that a floor plan is not to be determined, the routine continues to box 688 to store the determined one or more room shapes and / or the generated mapping information and / or other generated information, optionally provide some or all of the information to one or more recipients (e.g., to box 420 of routine 400 if called from box 420), and optionally further use some or all of the determined and generated information to provide a generated 2D floor plan and / or 3D computer model floor plan for display on one or more client devices and / or to one or more other devices, for automating navigation of such devices and / or associated vehicles or other entities, to similarly provide and use information about the determined room shapes and / or the linked set of images and / or additional information about the determined room contents and / or pathways between rooms, etc.

[0097] If it is determined in box 635 that the information or instructions received in box 605 are not to generate mapping information for the indicated building, the routine continues to box 690 to perform one or more other indicated operations as appropriate. Such other operations may include, for example: receiving and responding to requests for previously generated floor plans and / or previously determined room shapes and / or other generated information (e.g., requests for such information to be displayed on one or more client devices, requests for such information to be provided to one or more other devices for use in automatic navigation, etc.); obtaining and storing information about the building for use in subsequent operations (e.g., information about the size, number or type of rooms, total square footage, other adjacent or nearby buildings, adjacent or nearby vegetation, exterior images, etc.).

[0098] After frame 688 or 690, routine continues to frame 695, to determine whether to continue, for example, until receiving explicit instruction to terminate, or only when receiving explicit instruction to continue. If it is determined to continue, routine returns to frame 605 to wait and receive additional instructions or information, otherwise continues to frame 699 and ends.

[0099] Although not targeted FIG. 6A to FIG. 6BThe automatic operations shown in the exemplary embodiments of the present invention are described, but in some embodiments, a human user may further help facilitate some operations of the MIGM system, such as providing one or more types of input to an operator user and / or end user of the MIGM system, which are further used in subsequent automatic operations. As non-exclusive examples, such a human user may provide one or more types of input as follows: providing input to help link a set of images, such as providing input in box 625, which is used as part of the automatic operation of the box (e.g., specifying or adjusting the initial automatically determined orientation between one or more pairs of images, specifying or adjusting the initial automatically determined final global position of some or all images relative to each other, etc.); providing input in box 637, which is used as part of subsequent automatic operations, such as one or more of the information of the type shown about the building; providing input with respect to box 640, which is used as part of subsequent automatic operations to specify or adjust the initial automatically determined element positions and / or estimated room shapes, and / or manually combine information from multiple estimated room shapes of a room (e.g., to 6 and 683. The MIGM system may also include a plurality of inputs to a block 670 that are used as part of subsequent operations, such as specifying or adjusting an initial automatically determined location of a room shape within a generated floor plan and / or specifying or adjusting an initial automatically determined room shape within such floor plan itself; providing inputs to one or more of blocks 680 and 683 and 685 that are used as part of subsequent operations, such as specifying or adjusting initial automatically determined information of one or more types discussed with respect to those blocks; and / or specifying or adjusting initial automatically determined pose information (either initial pose information or subsequently updated pose information) for one or more of the panoramic images, etc. Additional details regarding embodiments in which a human user provides input are included elsewhere herein, which input is further used for additional automatic operations of the MIGM system.

[0100] FIG. 7A to FIG. 7B An exemplary embodiment of a flow chart for a building information viewer system routine 700 is shown, such as may be implemented by an embodiment of a BFPGLDP client application in some embodiments and situations (e.g., to provide feedback to a user of a capture device). The routine may be implemented by, for example, executing Figure 3The BFPGLDP client application 154 of the mobile device 185 and / or such client application or other building information viewer system executed on the mobile device 175 and / or other computing system or device is executed to receive and present building information (e.g., individual images; floor plans and / or other map-related information, such as determined room structure layout / shape, virtual tours of linked images, etc.; generated building description information; videos, etc.). FIG. 7A to FIG. 7B In the example of , the information presented is for one or more buildings (e.g., the interior of a house), but in other embodiments, other types of mapping information may be presented for other types of buildings or environments and used in other ways, as discussed elsewhere herein.

[0101] The illustrated implementation of the routine begins at block 705, where instructions or information are received. At block 710, the routine determines whether the instructions or information received at block 705 will present the determined information for one or more target buildings, and if so, continues to block 715 to determine whether the instructions or information received at block 705 will select one or more target buildings using specified criteria (e.g., based at least in part on the indicated buildings), and if not, continues to block 720 to obtain an indication of a target building to use from a user (e.g., based on a current user selection, such as from a displayed list or other user selection mechanism; based on information received in block 705; etc.). Otherwise, if it is determined in block 715 that one or more target buildings are selected from the specified criteria, the routine continues to block 725 where the routine obtains an indication of one or more search criteria to use, such as from the current user selection or as indicated in information or instructions received in block 705, and then searches stored information about buildings (e.g., floor plans, videos, generated text descriptions, etc.) to determine one or more buildings that meet the search criteria, or otherwise obtains an indication of one or more such matching target buildings, such as information currently or previously generated by the BFPGLDP system (such as information about the target buildings). FIG. 4A to FIG. 4BAn example of the operation of such a system is further discussed, and the BFPGLDP system is optionally called in box 720 to obtain such information). In the illustrated embodiment, the routine then further optionally selects the best matching target building from the one or more determined target buildings (e.g., the target building with the highest similarity or other match level for the specified criteria, or using another selection technique indicated in the instructions or other information received in box 705), while in other embodiments, the routine may alternatively present information for multiple target buildings that meet the search criteria (e.g., in a ranked order based on the degree of match; in a sequential manner, such as presenting one or more videos for each of the multiple buildings in a sequential manner, in a simultaneous manner, such as on a map of the surrounding area, etc.), and receive a user selection of the best matching target building from the multiple candidate target buildings.

[0102] After box 720 or 725, the routine continues to box 730 to determine whether the instruction or other information received in box 705 indicates to present one or more maps with one or more visual indicators for each of the one or more target buildings, and if so, continues to box 771 to do so, including retrieving information about the target buildings, the information including the generated floor plan model for each target building and the associated absolute location data for each floor plan. After box 771, the routine continues to box 773 to retrieve or otherwise generate information about one or more maps of one or more areas that include the location of the target building (e.g., one or more maps that match the criteria specified or otherwise determined in the information of box 705, such as with respect to zoom level and / or map size, and optionally using preference information or other information specific to the recipient), and optionally including one or more images with visual data for at least some of the areas covered by the map (if available) (e.g., satellite images or other overhead images, street-level images of the target building, etc.). After box 773, the routine continues to box 775 to determine, for each target building, an area on the map corresponding to the absolute location data of the floor plan of the target building, and overlay the floor plan of each target building on the corresponding map area, including fitting the floor plan to the visual representation of the target building if there is a visual representation of the target building on the image whose visual data includes the map area. The routine can also determine which types of information associated with the floor plan are included in the overlaid visual representation of the floor plan, such as based at least in part on the zoom level of the map and / or the size of the overlaid visual representation of the floor plan, and optionally using preference information or other information specific to the user recipient (e.g., based on one or more previous selections). In box 777, the routine selects a current view of the map including the overlaid floor plan visual representation (e.g., to select a zoom level, a map subset, etc.), and continues to box 779 to display or otherwise present the map with the overlaid floor plan visual representation in a GUI. The routine then waits for a user selection (or optionally waits for a timeout) in box 781, and if it is determined in box 783 that the user selection corresponds to adjusting the current view of the current map, the routine continues to box 785 to update the current view based on the user selection, and then returns to box 779 to update the displayed or otherwise presented information accordingly.The user selection and corresponding update of the current view may include, for example, displaying or otherwise presenting a piece of associated link information selected by the user (e.g., one or more types of additional or different building information for one or more target buildings, such as in response to the user's selection of a visual representation of a particular target building and / or selection of one or more other user controls; one or more types of additional or different neighborhood or other environmental information about one or more target buildings, such as in response to the user's selection of a visual representation of a particular target building and / or selection of one or more other user controls, etc.), and / or changing how the current view is displayed (e.g., zooming in or out; rotating and / or translating an area of ​​a displayed map, etc.). If it is determined in block 783 that the user has selected not to display further information of the current target building and / or map (e.g., displaying information of another building and / or map, ending the current display operation, etc.), or if the wait in block 781 has a timer expiration, the routine continues to block 795, and if the user selection involves such a further operation, returns to block 705 to perform the operation selected by the user.

[0103] Place

[0104] If it is determined in box 730 that the instruction or other information received in box 705 does not indicate that one or more building floor plans are to be presented on the map, the routine continues to box 735 to retrieve information for the target building for display (e.g., floor plans; other generated mapping information for the building, such as a set of linked images used as part of a virtual tour; generated building description information, etc.), and optionally associated link information indicating surrounding locations for the building interior and / or building exterior, and / or information about one or more generated interpretations or other descriptions of the target building, and select an initial view of the retrieved information (e.g., a floor plan, a view of a specific room shape, a specific image, some or all of the generated building description information, etc.). In box 740, the routine then displays or otherwise presents the current view of the retrieved information in the GUI, and waits for a user selection (or optionally a timeout) in box 745. After the user selection in block 745, if it is determined in block 750 that the user selection corresponds to adjusting the current view of the current target building (e.g., changing one or more aspects of the current view), the routine continues to block 755 to update the current view in accordance with the user selection and then returns to block 740 to update the displayed or presented information accordingly. The user selection and corresponding update of the current view may include, for example: displaying or otherwise presenting a piece of associated link information selected by the user (e.g., a particular image associated with the displayed visual indication of the determined acquisition location so that the associated link information is overlaid on at least some of the previously displayed; a particular other image that is linked to the current image and selected from the current image using a user-selectable control superimposed on the current image to represent the other image, etc.); and / or changing how the current view is displayed (e.g., zooming in or out; rotating the information if appropriate; selecting a new portion of the plan view to be displayed or otherwise presented, such as some or all of the new portion was not previously visible, or instead the new portion is a subset of the previously visible information, etc.). If it is determined in box 750 that the user chooses not to display further information about the current target building (e.g., display information for another building, end the current display operation, etc.), or if the wait in box 745 has a timer expired, the routine continues to box 795, and if the user selection involves such further operation, returns to box 705 to perform the operation selected by the user.

[0105] If it is determined in box 710 that the instruction or other information received in box 705 does not present information representing a building, the routine continues to box 760 to determine whether the instruction or other information received in box 705 indicates to identify other images (if any) corresponding to the one or more indicated target images, and if so, continues to boxes 762-764 to perform such activities. In particular, in box 762, the routine receives an indication of one or more target images for matching (e.g., from the information received in box 705 or based on one or more current interactions with the user) and one or more matching criteria (e.g., the amount of visual overlap), and in box 764, identifies one or more other images (if any) that match the indicated one or more target images, such as by interacting with the IDCA and / or MIGM system to obtain the one or more other images. Then, the routine displays or otherwise provides information about the identified one or more other images in box 764, such as providing information about them as part of the search results in a GUI to display one or more of the identified one or more other images, etc. If it is determined in block 760 that the instructions or other information received in block 705 are not to identify other images corresponding to one or more indicated target images, the routine continues to block 766 to determine whether the instructions or other information received in block 705 correspond to obtaining and providing guided acquisition instructions for one or more indicated target images (e.g., recently acquired images) during the image acquisition session. If so, continue to block 768, otherwise continue to block 790. In block 768, the routine obtains information about one or more types of guided acquisition instructions, such as by interacting with the IDCA system, and in block 768 displays or otherwise provides information about the guided acquisition instructions in the GUI, such as by superimposing the guided acquisition instructions on the partial plan view and / or the recently acquired image in a manner discussed in more detail elsewhere herein.

[0106] In box 790, the routine continues to perform other indicated operations as appropriate, such as configuring parameters to be used in various operations of the system (e.g., based at least in part on information specified by a user of the system (e.g., a user of a mobile device acquiring one or more buildings, an operator user of the BFPGLDP, and / or the MIGM system), etc.), including for personalizing the display of information for a particular recipient user based on his / her preferences or other information specific to the particular recipient user, to obtain and store other information about users of the system (e.g., preferences or other information specific to the user), to respond to requests for generated and stored information, to perform any housekeeping tasks, etc.

[0107] After box 764 or box 768 or box 790, or if it is determined in box 750 that the user selection does not correspond to the current building (or a timeout occurs), or if it is determined in box 783 that the user selection does not correspond to the current map (or a timeout occurs), the routine proceeds to box 795 to determine whether to continue, for example, until an explicit indication to terminate is received, or only when an explicit indication to continue is received. If it is determined to continue (including whether the user makes a selection related to a new building to be presented in box 745), the routine returns to box 705 to wait for additional instructions or information (or if the user makes a selection related to a new building to be presented in box 745, then directly proceed to box 730), and if not, proceed to step 799 and end.

[0108] Non-exclusive exemplary embodiments described herein are further described in the following clauses.

[0109] A01 (Method 26). A computer-implemented method for one or more computing devices to perform automated operations, comprising:

[0110] During the acquisition session, acquiring, by the one or more computing devices, a house having a plurality of rooms, and for each of the plurality of rooms:

[0111] a first panoramic image captured by a camera device moved by a first user and lacking any GPS (Global Positioning System) receiver, the first panoramic image captured at a first acquisition location in the room and having first visual data, the first visual data having 360 degrees of horizontal visual coverage showing walls of the room, second data simultaneously captured by a mobile capture device moved independently from the camera device by a second user, the second data comprising a plurality of GPS location data points for a plurality of second capture locations in the room;

[0112] determining, by the one or more computing devices, a floor plan for the house having associated GPS location data based at least in part on combining the second data captured at the plurality of second capture locations in each of the plurality of rooms with information about the room determined from analysis of the first visual data of the first panoramic image captured in the room, the method comprising:

[0113] analyzing, by the one or more computing devices and for each of the plurality of rooms, first visual data of a first panoramic image captured in the room to determine a three-dimensional (“3D”) room shape of the room that models at least some walls of the room as flat surfaces, and a position within the determined 3D room shape of a first acquisition location in the room;

[0114] determining, by the one or more computing devices and for each of the plurality of rooms, a representative GPS location data point associated with the first acquisition location in the room, including identifying two or more candidate GPS location data points from the plurality of GPS location data points captured in the room using capture times of the two or more candidate GPS location data points, and using the two or more candidate GPS location data points to generate the representative GPS location data point for the first acquisition location in the room;

[0115] generating, by the one or more computing devices, the floor plan of the house, including positioning the determined 3D room shapes of the plurality of rooms relative to each other and determining a position of the first acquisition location in each of the plurality of rooms in the floor plan using a local coordinate system of the floor plan;

[0116] determining, by the one or more computing devices, a global transformation that maps data in a first data set to data in a second data set, the first data set comprising locations in a floor plan of the first acquisition location of the plurality of rooms using a local coordinate system of the floor plan, and the second data set comprising representative GPS location data points associated with the first acquisition location;

[0117] determining, by the one or more computing devices and for each of the plurality of rooms, GPS position data for at least some walls of the determined 3D room shape of the room by combining the determined GPS position data for the first acquisition location in the room with the determined position within the determined 3D room shape of the first acquisition location in the room using the global transform; and

[0118] identifying, by the one or more computing devices, at least one exterior wall of the house represented on the floor plan, and determining associated GPS location data for the at least some walls of the determined 3D room shapes for the plurality of rooms; and

[0119] The generated floor plan for the house is displayed, by the one or more computing devices and using the associated GPS location of the at least one exterior wall for the floor plan, the floor plan being located on a map of an area surrounding the property where the house is located.

[0120] A02. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0121] obtaining, by the one or more computing devices, a plurality of images and a plurality of GPS (Global Positioning System) location data points in a plurality of rooms of a building, wherein, for each of the plurality of rooms, one or more of the plurality of images comprises one or more of the plurality of images acquired by a camera device at a first acquisition position in the room and having visual coverage of a wall of the room, and a plurality of GPS location data points in the plurality of GPS location data points, the plurality of GPS location data points in the plurality of rooms being simultaneously captured at a plurality of second acquisition positions in the plurality of GPS location data points provided by a mobile capture device that moves independently of the camera device;

[0122] obtaining, by the one or more computing devices and based on visual data of the plurality of images, a floor plan of a building and information about the position of the first acquisition location on the floor plan using a coordinate system local to the floor plan, including determining, for each of the plurality of rooms, a room shape of the room, the room shape of the room including at least some walls of the room and a position within the room shape of the first acquisition location in the room, and also including positioning the room shapes of the plurality of rooms relative to each other;

[0123] determining, by the one or more computing devices and for each of the plurality of rooms, a representative GPS location data point associated with the first acquisition location in the room, including identifying two or more candidate GPS location data points from the plurality of GPS location data points captured in the room using capture times of the two or more candidate GPS location data points, and using the two or more candidate GPS location data points to generate the representative GPS location data point for the first acquisition location in the room;

[0124] determining, by the one or more computing devices, a global transformation that maps data in a first data set to data in a second data set, the first data set comprising locations in a floor plan of the first acquisition location using a coordinate system local to the floor plan, and the second data set comprising representative GPS location data points associated with the first acquisition location;

[0125] determining, by the one or more computing devices, one or more GPS location data points for one or more locations on the floor plan corresponding to one or more locations on an exterior wall of the building, including, for the one or more locations on the one or more exterior walls, transforming, using the global transformation, local coordinates of the one or more GPS location data points for the one or more locations in a coordinate system local to the floor plan; and

[0126] A visual representation of a floor plan of the building overlaid on a map is provided by the one or more computing devices at map locations corresponding to the one or more GPS location data points determined to be located on the one or more exterior walls of the building.

[0127] A03. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0128] obtaining, by the one or more computing devices, after obtaining a plurality of images and a plurality of GPS (Global Positioning System) location data points in a plurality of rooms of the building, at least a partial floor plan of the building based at least in part on visual data of the plurality of images, wherein the plurality of images include, for each of the plurality of rooms, an image of the plurality of images acquired by a camera device at a first acquisition location in the room and having the following visual overlay: the plurality of GPS location data points include, for each of the plurality of rooms, a plurality of GPS location data points of the plurality of GPS location data points captured at a plurality of second capture locations in the room, the plurality of second capture locations being separate from and captured within a defined amount of time of the capture of the images in the room, wherein the at least partial floor plan includes determined room shapes of the plurality of rooms positioned relative to each other, each room shape for a room including a representation of at least some walls of the room, and wherein the at least partial floor plan includes information about a location on the at least partial floor plan of the first acquisition location using a coordinate system local to the at least partial floor plan;

[0129] determining, by the one or more computing devices, and for each of the plurality of rooms, a representative GPS location data point associated with the first capture location in the room by using the plurality of GPS location data points captured at the plurality of second capture locations in the room;

[0130] determining, by the one or more computing devices, one or more GPS location data points for one or more locations on the at least partial floor plan, including, for local coordinates of the one or more locations in a coordinate system local to the at least partial floor plan, converting those local coordinates to the one or more GPS location data points; and

[0131] A visual representation of the at least partial floor plan for the building is provided, by the one or more computing devices, for inclusion on a map at one or more map locations corresponding to the one or more GPS location data points of the determined one or more locations.

[0132] A04. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0133] obtaining a plurality of images and a plurality of absolute position data points in one or more rooms of a building, including, for each of the one or more rooms, an image in the plurality of images acquired by a camera device at each of one or more first acquisition locations in the room and having visual coverage of at least some walls of the room, and a plurality of absolute position data points in the plurality of absolute position data points captured by a mobile capture device at a plurality of second capture locations in the room that is movable independently of the camera device;

[0134] obtaining, based at least in part on the visual data of the plurality of images and for each of the one or more rooms, a room shape of the room and information about a position in the room shape of each of the one or more first acquisition positions in the room, the first acquisition positions using a coordinate system local to the room shape, the room shape of the room including representations of the at least some walls of the room;

[0135] for each of the one or more rooms, determining a representative absolute position data point associated with each of the one or more first capture locations in the room by using the plurality of absolute position data points captured at the plurality of second capture locations in the room;

[0136] determining a global transformation that maps data in a first data set to data in a second data set, the first data set comprising, for each of the one or more rooms, a location in the room of the one or more first acquired locations in the room using a coordinate system local to a room shape of the room, and the second data set comprising a representative GPS location data point associated with the first acquired location;

[0137] determining, for each of at least one room shape of the one or more rooms, one or more absolute position data points for one or more locations on the room shape, including using the global transformation, for local coordinates of the one or more locations, converting those local coordinates of the one or more locations to the one or more absolute position data points, the one or more locations being in a local coordinate system of the room shape;

[0138] A visual representation of the at least one room shape is provided using the determined one or more absolute position data points for each of the at least one room shape.

[0139] A05. A computer-implemented method as described in any of clauses A01-A04, wherein the two or more candidate GPS location data points associated with the first acquisition location in one of the multiple rooms are acquired at multiple times and at multiple acquisition locations and are a subset of the multiple GPS location data points acquired in the one room after performing a smoothing operation to remove outlier data points, wherein the one or more computing devices include the mobile acquisition device. And wherein the determination of the representative GPS location data point associated with the first acquisition location in the one room comprises:

[0140] Retrieving, by the one or more computing devices, a capture time of the first panoramic image captured at the first capture location in the one room;

[0141] determining, by the one or more computing devices, the two or more candidate GPS location data points associated with the first acquisition location in the one room based at least in part on the two or more candidate GPS location data points, the two or more candidate GPS location data points having capture times within a defined time window around the retrieved acquisition time; and

[0142] The representative GPS location data point associated with the first acquisition location in the one room is generated by the one or more computing devices based at least in part on average location data of multiple acquisition locations of the two or more candidate GPS location data points associated with the first acquisition location in the one room.

[0143] A06. A computer-implemented method as described in any of clauses A01-A05, wherein determining GPS location data for at least some of the walls of the determined 3D room shape for each of the plurality of rooms comprises, by the one or more computing devices and for each of at least some of the first acquisition locations, determining a revised representative GPS location data point for the first acquisition location based on the global transformation, and further comprising extending the GPS location data from each of the one or more first acquisition locations to at least some of the walls of the determined 3D room shape, wherein the first acquisition locations are located in the room, wherein the walls to which the GPS location data is extended include the at least one exterior wall.

[0144] A07. A computer-implemented method as described in any of clauses A01-A06, wherein the determination of the global transformation comprises at least one of:

[0145] determining, by one or more computing devices, a rigid transformation between the first and second data sets using a Kabsch algorithm; or

[0146] determining, by one or more computing devices, a rigid transformation between the first and second data sets using a quaternion estimation algorithm; or

[0147] Determining, by one or more computing devices, a rigid transformation between the first and second data sets using an algorithm capable of solving the Wahba problem; or

[0148] determining, by the one or more computing devices, a global transformation using a point set registration technique; or

[0149] The global transformation is determined by the one or more computing devices using a trained machine learning model that takes at least the first data set and the second data set as input and produces the global transformation as output.

[0150] A08. A computer-implemented method as described in any of clauses A01-A07, wherein the two or more candidate GPS location data points associated with the first capture location in one of the multiple rooms are captured at multiple times and multiple capture locations, and the two or more candidate GPS location data points are a subset of the multiple GPS location data points captured in the one room after performing a smoothing operation to remove outlier data points, wherein the camera device lacks any GPS receiver. And wherein the determination of the representative GPS location data point associated with the first acquisition location in the one room comprises:

[0151] retrieving, by the one or more computing devices, a capture time of one of the images captured at the first capture location in the one room;

[0152] determining, by the one or more computing devices, the two or more candidate GPS location data points associated with the first acquisition location in the one room based at least in part on the two or more candidate GPS location data points, the two or more candidate GPS location data points having capture times within a defined time window around the retrieved acquisition time; and

[0153] The representative GPS location data point associated with the first acquisition location in the one room is generated by the one or more computing devices based at least in part on average location data of multiple acquisition locations of the two or more candidate GPS location data points associated with the first acquisition location in the one room.

[0154] A09. A computer-implemented method as described in any of clauses A01-A08, wherein determining the one or more GPS location data points of the one or more locations on the floor plan corresponding to the one or more locations on the exterior wall of the building includes: determining, by the one or more computing devices, a revised representative GPS location data point for each of at least some of the first acquisition locations based on the global transformation, and extending the GPS location data from each of the one or more first acquisition locations to one or more walls of a room shape of a room in which the first acquisition location is located, wherein the walls to which the GPS location data is extended include one or more exterior walls.

[0155] A10. A computer-implemented method as described in any of clauses A01-A09, wherein the determination of the global transformation comprises at least one of:

[0156] determining, by one or more computing devices, a rigid transformation between the first and second data sets using a Kabsch algorithm; or

[0157] determining, by one or more computing devices, a rigid transformation between the first and second data sets using a quaternion estimation algorithm; or

[0158] Determining, by one or more computing devices, a rigid transformation between the first and second data sets using an algorithm capable of solving the Wahba problem; or

[0159] determining, by the one or more computing devices, a global transformation using a point set registration technique; or

[0160] The global transformation is determined by the one or more computing devices using a trained machine learning model that takes at least the first data set and the second data set as input and produces the global transformation as output.

[0161] A11. A computer-implemented method as described in any of clauses A01-A10, wherein the stored content includes software instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform further automated operations, the further automated operations including determining a global transformation by the one or more computing devices, the global transformation mapping data in a first data set to data in a second data set, the first data set including locations in the at least partial floor plan of the first acquisition location, the locations locally using the at least partial floor plan using the coordinate system and the second data set including representative GPS location data points associated with the first acquisition location, and wherein converting local coordinates in the local coordinate system of the at least partial floor plan to the one or more GPS location data points includes using the global transformation.

[0162] A12. A computer-implemented method as described in clause A11, wherein the determination of the global transformation includes at least one of the following:

[0163] determining, by one or more computing devices, a first rigid transformation between the first and second data sets using a Kabsch algorithm; or

[0164] determining, by one or more computing devices, a second rigid transformation between the first and second data sets using a quaternion estimation algorithm; or

[0165] determining, by one or more computing devices, a third rigid transformation between the first and second data sets using an algorithm that solves the Wahba problem; or

[0166] determining, by the one or more computing devices, a global transformation using a point set registration technique; or

[0167] The global transformation is determined by the one or more computing devices using a trained machine learning model that takes at least the first data set and the second data set as input and produces the global transformation as output.

[0168] A13. A computer-implemented method as described in any of clauses A01-A12, wherein the stored content includes software instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform further automated operations, the further automated operations comprising determining, by the one or more computing devices, a global transformation that maps data in a first data set to data in a second data set, the first data set comprising locations using the coordinate system in the at least partial floor plan of the first acquisition location, the at least partial floor plan and the second data set comprising representative GPS location data points associated with the first acquisition location, and wherein converting local coordinates in a coordinate system local to the at least partial floor plan to the one or more GPS location data points comprises:

[0169] determining, by the one or more computing devices, a revised representative GPS location data point for each of the at least some of the first acquired locations based on the global transformation; and

[0170] The one or more computing devices extend GPS location data from each of one or more of the at least some first acquisition locations to one or more walls of a room shape of a room in which the first acquisition locations are located, wherein the walls to which the GPS location data is extended include one or more exterior walls of the building, and the one or more exterior walls have the one or more locations on the at least partial floor plan.

[0171] A14. A computer-implemented method as described in any of clauses A01-A13, wherein the visual representation of the at least portion of the floor plan is at least one of a two-dimensional floor plan model or a three-dimensional floor plan model, and wherein determining the one or more GPS location data points for the one or more locations on the at least portion of the floor plan includes determining the one or more GPS location data points for one or more locations corresponding to one or more exterior walls of the building.

[0172] A15. A computer-implemented method as described in any of clauses A01-A14, wherein capturing the multiple GPS location data points at the multiple second locations in one of the multiple rooms is performed by a mobile capture device that moves independently of the camera device and includes capturing additional GPS location data points in the one room at additional times, and wherein the automated operation also includes selecting the multiple GPS location data points for determining the representative GPS location data of the first acquisition location in the one room in at least the following steps: a portion of the multiple GPS location data points captured within a specified amount of time to acquire an image in the one room.

[0173] A16. The computer-implemented method of clause A15, wherein the camera device lacks any GPS receiver, wherein the automatic operation further comprises, prior to selecting the plurality of GPS location data points, performing a smoothing operation on the additional GPS location data points, the smoothing operation comprising removing one or more of the additional GPS location data points determined to be outliers, and wherein the determining of the representative GPS location data point associated with the first acquisition location in the one room comprises:

[0174] Retrieving, by the one or more computing devices, a capture time of an image captured at a first capture location in the one room;

[0175] selecting, by the one or more computing devices, the plurality of GPS location data points associated with the first acquisition location in the one room based at least in part on the plurality of GPS location data points having capture times within a defined time window around the retrieved acquisition time, the defined time window being based on the defined amount of time; and

[0176] A representative GPS location data point associated with a first capture location in the one room is generated by the one or more computing devices based at least in part on combining location data of a plurality of GPS location data points of a plurality of second capture locations in the one room.

[0177] A17. A computer-implemented method as described in any of clauses A01-A16, wherein determining the representative GPS location data point associated with the first acquisition location in one of the plurality of rooms comprises: selecting, by the one or more computing devices, one of the plurality of GPS location data points captured in the one room to use as the representative GPS location data point associated with the first acquisition location in the one room. The selection is based at least in part on a time when the one GPS location data point was captured and a time when the image was acquired in the one room.

[0178] A18. A computer-implemented method as described in any of clauses A01-A17, wherein capturing the plurality of GPS location data points at the plurality of second capture locations in one of the plurality of rooms is performed by the camera device and includes capturing additional GPS location data points in the one room at additional times, and wherein the automated operation further includes selecting the plurality of GPS location data points for use in determining the representative GPS location data for the first acquisition location in the one room based at least in part on the plurality of GPS location data points captured within a specified amount of time for acquiring images in the one room.

[0179] A19. A computer-implemented method as described in any of clauses A01-A18, wherein obtaining the at least partial floor plan includes determining a room shape of the room for each of the multiple rooms, the room shape including at least some walls of the room and the position within the room shape of the first acquisition location in the room, and also includes positioning the room shapes of the multiple rooms relative to each other.

[0180] A20. A computer-implemented method as described in any of clauses A01-A19, wherein providing the visual representation of the at least partial floor plan for the building includes overlaying the visual representation of the at least partial floor plan by the one or more computing devices at the one or more map locations on the map, the one or more map locations corresponding to the one or more GPS location data points determined for the one or more locations. The map covering an area larger than the property on which the building is located also includes sending the map with the overlaid visual representation to a client device for display.

[0181] A21. A computer-implemented method as described in any of clauses A01-A20, wherein the automated operation further comprises:

[0182] acquiring, by the one or more computing devices, a plurality of additional GPS location data points of the exterior of the building, the additional GPS location data points being captured by a data capture device on the property where the building is located; and

[0183] At least one of the plurality of additional GPS location data points is determined, by the one or more computing devices, to represent an exterior doorway of the building, the determination being based on at least one of analyzing additional visual data including one or more additional acquired images of the exterior doorway to determine when the data capture device is proximate to the exterior doorway or receiving input from an operator user of the data capture device regarding positioning of the data capture device proximate to the exterior doorway.

[0184] And wherein the one or more GPS location data points determined for the one or more locations on the at least partial floor plan, or at least one of the visual representations of the at least partial floor plan for the building located on the map is also based in part on at least one additional GPS location data point determined to represent the exterior doorway.

[0185] A22. A computer-implemented method as described in any of clauses A01-A21, wherein the automated operation further comprises:

[0186] acquiring, by the one or more computing devices, a plurality of additional GPS location data points of the exterior of the building, the plurality of additional GPS location data points being captured by a data capture device on a property where the building is located, wherein the plurality of additional GPS location data points are captured during a traversal of an exterior boundary of the building; and

[0187] determining, by the one or more computing devices, at least one of a shape or a position of an exterior boundary of a building based at least in part on the plurality of additional GPS location data points,

[0188] And wherein determining the one or more GPS location data points for the one or more locations on the at least partial floor plan, or locating on the map at least one of the visual representations of the at least partial floor plan for the building is also based in part on at least one of the determined shape or the position of the external boundary.

[0189] A23. A computer-implemented method as described in any of clauses A01-A22, wherein the automated operation further comprises:

[0190] obtaining, by the one or more computing devices, a plurality of additional GPS location data points of the exterior of the building, the plurality of additional GPS location data points being captured by a data capture device on a property where the building is located, wherein the plurality of additional GPS location data points are captured during a traversal of an exterior walkway to the building and correspond to substantially straight lines; and

[0191] determining, by the one or more computing devices, a location of the exterior walkway based at least in part on the plurality of additional GPS location data points;

[0192] And wherein determining the one or more GPS location data points for the one or more locations on the at least partial floor plan, or locating at least one of the visual representations of the at least partial floor plan of the building on the map is also based in part on the determined location of the exterior walkway.

[0193] A24. A computer-implemented method as described in any of clauses A01-A23, wherein the plurality of GPS location data points are captured in the plurality of rooms by a data capture device, and wherein the automatic operation further comprises, before obtaining the plurality of GPS location data points in the plurality of rooms:

[0194] providing, by the one or more computing devices, instructions related to the movement pattern of the data capture device to the exterior of the building so as to enable an increase in the accuracy of the plurality of GPS location data points in the plurality of rooms during subsequent capture of the plurality of GPS location data points in the plurality of rooms; and

[0195] A plurality of additional GPS location data points captured by the data capture device corresponding to movement patterns of the data capture device outside of the building are obtained by the one or more computing devices.

[0196] A25. A computer-implemented method as described in any of clauses A01-A24, wherein the automated operation also includes obtaining, by the one or more computing devices, at least one of additional geographic directional data for at least one location at the building or additional absolute position data for the non-GPS position data of the at least one location, and wherein positioning the visual representation of the at least partial floor plan of the building on the map is based in part on the at least one of the additional geographic directional data or the additional absolute position data.

[0197] A26. A computer-implemented method as described in any of clauses A01-A25, wherein the images acquired in the multiple rooms each include 360 ​​degrees of horizontal visual coverage around a vertical axis, and wherein the determination of the room shapes of the multiple rooms is performed without using any depth information from any depth sensing sensor, wherein the depth information is used to determine the distance from the position of the camera device in the multiple rooms to surrounding surfaces.

[0198] A27. A computer-implemented method as described in any of clauses A01-A26, wherein the absolute position data points are GPS (Global Positioning System) data points, and wherein the one or more locations on each of the at least one room shapes include one or more locations corresponding to one or more exterior walls of the building.

[0199] A28. A computer-implemented method as described in any of clauses A01-A27, wherein the plurality of images and the plurality of absolute position data points include images and absolute position data points from a plurality of rooms of the building, and wherein obtaining a room shape for each of the one or more rooms comprises:

[0200] for each of a plurality of rooms, determining a room shape of the room including at least some walls of the room and a position within the room shape of a first acquisition location in the room;

[0201] positioning room shapes of the plurality of rooms relative to each other to form at least a partial floor plan for the building;

[0202] And wherein the visual representation of at least one room shape provided is a visual representation of the at least partial floor plan, and wherein the local coordinate system of the room shape of each of the one or more rooms is a local coordinate system of the at least partial floor plan.

[0203] A29. The computer-implemented method of clause A28, wherein providing the visual representation of the at least one room shape for the building comprises: overlaying the visual representation of the at least partial floor plan on a map at the one or more map locations corresponding to the determined one or more absolute position data points for each of the at least one room shape, the map covering an area larger than the property on which the building is located, and further comprising sending the map with the overlaid visual representation to a client device for display.

[0204] A30. The computer-implemented method of any of clauses A01-A29, wherein the camera device lacks any GPS receiver, and wherein determining a representative absolute position data point associated with a first acquisition location in a room comprises:

[0205] Determining a capture time of the image captured at the first capture location;

[0206] selecting two or more of the plurality of absolute position data points acquired in the room based at least in part on the two or more absolute position data points having a capture time within a defined amount of time of the determined acquisition time; and

[0207] A representative absolute position data point associated with the first acquired position is generated based at least in part on combining the position data of the two or more absolute position data points.

[0208] A31. A computer-implemented method comprising the steps of performing a plurality of automated operations that implement techniques substantially as disclosed herein.

[0209] B01. A non-transitory computer-readable medium having stored executable software instructions and / or other storage contents, wherein the stored executable software instructions and / or other storage contents enable one or more computing systems to perform automatic operations to implement the method of any of clauses A01-A31.

[0210] B02. A non-transitory computer-readable medium having stored executable software instructions and / or other stored content, which enables one or more computing systems to perform automatic operations that implement the technology substantially as disclosed herein.

[0211] C01. One or more computing systems comprising one or more hardware processors and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations implementing the method of any of clauses A01-A31.

[0212] CO2. One or more computing systems comprising one or more hardware processors and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations that implement techniques substantially as disclosed herein.

[0213] D01. A computer program adapted to perform the method of any of clauses A01-A31 when said computer program is run on a computer.

[0214] E01. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0215] During the acquisition session, acquiring, by the one or more computing devices, a house having a plurality of rooms, and for each of the plurality of rooms:

[0216] a first panoramic image captured by a camera device moved by a first user and lacking any GPS (Global Positioning System) receiver, the first panoramic image captured at a first acquisition location in the room and having first visual data, the first visual data having 360 degrees of horizontal visual coverage showing walls of the room

[0217] second data simultaneously captured by a mobile capture device that moves independently of the camera device, the second data comprising a plurality of GPS location data points for a plurality of second capture locations in the room;

[0218] determining, by the one or more computing devices, a floor plan for the house having associated GPS location data based at least in part on combining the second data captured at the plurality of second capture locations in each of the plurality of rooms with information about the room determined from analysis of the first visual data of the first panoramic image captured in the room, the method comprising:

[0219] analyzing, by the one or more computing devices and for each of the plurality of rooms, first visual data of a first panoramic image captured in the room to determine a three-dimensional (“3D”) room shape of the room that models at least some walls of the room as flat surfaces, and a position within the determined 3D room shape of a first acquisition location in the room;

[0220] determining, by the one or more computing devices and for each of the plurality of rooms, a representative GPS location data point associated with the first acquisition location in the room, including identifying two or more candidate GPS location data points from the plurality of GPS location data points captured in the room using capture times of the two or more candidate GPS location data points, and using the two or more candidate GPS location data points to generate the representative GPS location data point for the first acquisition location in the room;

[0221] generating, by the one or more computing devices, the floor plan of the house, including positioning the determined 3D room shapes of the plurality of rooms relative to each other and determining a position of the first acquisition location in each of the plurality of rooms in the floor plan using a local coordinate system of the floor plan;

[0222] determining, by the one or more computing devices and for each of the plurality of rooms, GPS position data for at least some walls of the determined 3D room shape of the room by combining the determined GPS position data for the first acquisition location in the room with the determined position within the determined 3D room shape of the first acquisition location in the room; and

[0223] identifying, by the one or more computing devices, at least one exterior wall of the house represented on the floor plan, and determining associated GPS location data for the at least some walls of the determined 3D room shapes for the plurality of rooms;

[0224] determining, by the one or more computing devices and based at least in part on the determined associated GPS location data for the at least one exterior wall, a location on a map of an area including a building corresponding to the floor plan;

[0225] determining, by the one or more computing devices, one or more levels of detail for an interior portion of the building based at least in part on the determined location on the map to display each of the one or more stories of the building on the map; and

[0226] Presenting, by the one or more computing devices and as part of a graphical user interface (GUI), the map with a visual representation of the floor plan, including overlaying the visual representation of the floor plan on the map at the determined location and fitting exterior boundaries of buildings on the floor plan to existing visual representations of buildings on the map prior to overlaying, and wherein the overlaid visual representation includes visual indications for interior portions of the building having the determined one or more levels of detail and including at least interior walls of the plurality of rooms.

[0227] E02. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0228] obtaining, by the one or more computing devices, a floor plan for a building having a plurality of rooms and information about a plurality of GPS (Global Positioning System) location data points captured during a traversal of interiors of the plurality of rooms, the floor plan including room shapes of the plurality of rooms positioned relative to one another;

[0229] determining, by the one or more computing devices and based at least in part on the plurality of GPS location data points, one or more GPS location data points for one or more locations on the floor plan corresponding to at least one exterior wall of the building, including extending one or more of the captured GPS data points from the interior of the plurality of rooms to the at least one exterior wall of the building;

[0230] determining, by the one or more computing devices, a location on a map of an area including a building corresponding to the floor plan based at least in part on the determined one or more GPS location data points for the one or more locations on the floor plan;

[0231] determining, by the one or more computing devices, one or more levels of detail for an interior portion of the building based at least in part on the determined location on the map to display each of the one or more stories of the building on the map; and

[0232] A map having a visual representation of a floor plan is presented by the one or more computing devices and as part of a graphical user interface (GUI), the floor plan being overlaid on the map at the determined location and including a visual indication of the interior of the building, the floor plan having the determined one or more levels of detail and including at least interior walls of a plurality of rooms.

[0233] E03. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0234] obtaining, by the one or more computing devices, a floor plan for a building having a plurality of rooms and information regarding a plurality of GPS location data points captured during a traversal of the plurality of rooms, the floor plan including room shapes of the plurality of rooms positioned relative to one another;

[0235] determining, by the one or more computing devices and based at least in part on the plurality of GPS location data points, one or more GPS location data points for one or more locations on the floor plan corresponding to at least one exterior wall of the building;

[0236] determining, by the one or more computing devices, a location on a map of an area including a building corresponding to the floor plan based at least in part on the determined one or more GPS location data points for the one or more locations on the floor plan;

[0237] retrieving, by the one or more computing devices, additional information about the plurality of types of buildings; and

[0238] The map is presented by the one or more computing devices and as part of a graphical user interface (GUI), the map having a visual representation of the floor plan at the determined location overlaid in a first portion of the GUI, and additional portions of the GUI, each of the additional portions of the GUI including at least some additional information for a corresponding type of the plurality of types.

[0239] E04. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0240] For a building having a plurality of rooms, obtaining at least a partial floor plan of the building, the floor plan including room shapes of one or more rooms of the plurality of rooms and information about a plurality of absolute position data points captured in the one or more rooms;

[0241] determining one or more absolute position data points for one or more locations on the at least partial floor plan based at least in part on the plurality of absolute position data points;

[0242] determining a location on a map of an area including a building corresponding to the at least partial floor plan based at least in part on the determined one or more absolute position data points of the one or more locations on the at least partial floor plan;

[0243] determining, for an interior portion of a building, one or more levels of detail for displaying each of the one or more stories of the building on a map; and

[0244] A visual representation of the at least partial floor plan is presented to the map, the at least partial floor plan being displayed at the determined location on the map and including data corresponding to the determined one or more levels of detail of an interior portion of the building.

[0245] E05. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0246] acquiring, by the one or more computing devices, for a building, at least a partial floor plan of the building including room shapes of one or more rooms of the building, and first information regarding a plurality of absolute position data points captured in the one or more rooms, and second information regarding one or more exterior features of the property and exterior of the building;

[0247] determining, by the one or more computing devices and based at least in part on the plurality of absolute position data points, one or more absolute position data points for one or more locations on the at least partial plan view;

[0248] determining, by the one or more computing devices, a location corresponding to the at least partial floor plan on a map including the property based at least in part on the determined one or more absolute position data points for the one or more locations on the at least partial floor plan;

[0249] determining, by the one or more computing devices and for each of the one or more external features, a location of an attribute with respect to the external feature; and

[0250] A visual representation of the at least portion of the floor plan shown at the determined location on the map is presented to the map by the one or more computing devices, and an additional visual representation of the external feature is also presented, for each of the one or more external features, at the determined location on an attribute of the external feature.

[0251] E06. A computer-implemented method as described in any of clauses E01-E05, wherein the at least partial floor plan includes multiple room shapes of two or more of the multiple rooms positioned relative to each other, and wherein presenting the visual representation of the at least partial floor plan includes presenting in a graphical user interface a visual representation of the at least partial floor plan overlaid on the map and having a visual indication of interior walls of the two or more rooms.

[0252] E07. The computer-implemented method of clause E06, wherein obtaining the at least partial floor plan and information about the plurality of absolute position data points comprises:

[0253] obtaining a plurality of images captured in two or more rooms of a building, including, for each of the one or two or more rooms, one or more of the plurality of images being acquired by a camera device at one or more first acquisition locations in the room and having visual coverage of at least some walls of the room, and wherein a plurality of the plurality of absolute position data points being captured at a plurality of second acquisition locations in the room by a mobile capture device that is independently movable from a camera device;

[0254] determining, for each of the two or more rooms and based on the visual data of the one or more images captured in the room, a room shape of the room including the at least some of the walls of the room; and

[0255] The at least partial floor plan is generated by positioning room shapes of the two or more rooms relative to each other.

[0256] E08. A computer-implemented method as described in any of clauses E01-E07, wherein determining the one or more levels of detail for displaying the interior portion of the building on the map for each of the one or more stories of the building comprises determining a size on the map at which the visual representation of the floor plan is to be displayed based on at least one of a determined location on the map or a zoom level of the map, and selecting multiple levels of detail to display for the interior portion of the building based at least in part on the determined size.

[0257] E09. A computer-implemented method according to claim E08, wherein the multiple levels of detail include two or more interior walls of the at least partial floor plan, or one or more types of appliances at indicated locations within the interior walls, or one or more types of fixtures at indicated locations on or within the interior walls, or one or more types of built-in structural elements at indicated locations on or within the interior walls. Or one or more types of mounting elements at indicated locations on or within the interior walls, or one or more types of furniture at indicated locations within the interior walls.

[0258] E10. A computer-implemented method as described in any of clauses E08-E09, wherein selecting the multiple levels of detail to be displayed for the interior portion of the building is also based at least in part on information specific to a user to whom the map is presented.

[0259] E11. A computer-implemented method as described in any of clauses E08-E10, wherein the building has a plurality of stories, wherein the visual representation of the at least partial floor plan includes a room shape of at least one room on each of the plurality of stories, and wherein the method further comprises:

[0260] receiving one or more user interactions with the rendered map after presenting the map having a visual representation of the at least partial floor plan including room shapes of the at least one room on each of the plurality of stories; and

[0261] In response to the one or more user interactions, the presented map is updated to display an updated visual representation of the at least partial floor plan, the updated visual representation including only information about a single story of the plurality of stories.

[0262] E12. A computer-implemented method as described in clause E11, wherein the one or more user interactions include changing a zoom level of the map, and the updating of the presented map includes selecting the single story based at least in part on the changed zoom level of the map.

[0263] E13. A computer-implemented method as described in any of clauses E08-E12, wherein the building has a plurality of stories, wherein determining an interior portion of the building to display the one or more levels of detail of each of the one or more stories of the building on the map further comprises selecting one of the plurality of stories, wherein the visual presentation of the at least partial floor plan comprises a room shape of at least one room on the selected one of the stories. And wherein the automated operation further comprises:

[0264] receiving one or more user interactions with the rendered map after presenting the map having a visual representation of the at least partial floor plan including a room shape for the at least one room on the selected one story; and

[0265] In response to the one or more user interactions, updating the presented map to display an updated visual presentation of the at least partial floor plan including additional room shapes for at least one additional room on an additional story of the plurality of stories different from the selected one.

[0266] E14. A computer-implemented method as described in any of clauses E01-E13, wherein the one or more user interactions include changing a zoom level of the map, and the updating of the presented map includes determining, based at least in part on the changed zoom level of the map, to add a combination of visual information of the additional story and visual information of the selected one story.

[0267] E15. A computer-implemented method as described in any of clauses E01-E14, wherein the automatic operation further includes collecting data about the environment of the building before presenting the visual representation to the map, and generating the visual representation of the at least portion of the floor plan to include at least some of the collected data as part of the visual representation.

[0268] E16. A computer-implemented method according to any one of claims E01-E15, wherein the automatic operation further comprises, before the map is presented together with the visual representation, collecting data about the environment of the building, and wherein the presentation of the map comprises overlaying additional information adjacent to the visual representation on the map to provide a visual indication of at least some of the collected data in a manner relative to the visual representation presented on the map.

[0269] E17. A computer-implemented method as described in any of clauses E01-E16, wherein the automatic operation also includes, before presenting the map with the visual representation, obtaining information about one or more additional buildings located near the building, and wherein determining the location on the map or presenting the map with the visual representation at least one includes positioning the visual representation relative to the location on the map of the one or more additional buildings.

[0270] E18. A computer-implemented method according to any of clauses E01-E17, wherein presenting the map with the visual representation comprises: overlaying the visual representation of the at least portion of the floor plan on the map using the determined location on the map, and further overlaying at least one additional visual indication of at least one geographic direction, the visual representation being at least one of a two-dimensional floor plan model or a three-dimensional floor plan model. And further comprising sending the map with the overlaid visual representation to a client device for display, the map having an area larger than the property on which the building is located.

[0271] E19. A computer-implemented method as described in any of clauses E01-E18, wherein the map includes an existing visual representation of the building prior to overlaying the visual representation of the at least partial floor plan, and wherein overlaying the visual representation of the at least partial floor plan includes fitting an outer boundary of the building on the at least partial floor plan to the existing visual representation of the building.

[0272] E20. A computer-implemented method according to any of clauses E01-E19, wherein the method further comprises determining, by the one or more computing devices and based at least in part on the determined location on the map, one or more levels of detail for an interior portion of the building to display each of the one or more stories of the building on the map, and wherein the presentation of the map with the visual representation of the floor plan comprises overlaying the visual representation at the determined location and the visual representation comprises visually indicating the interior portion of the building with the determined one or more levels of detail and comprising at least interior walls of a plurality of rooms.

[0273] E21. A computer-implemented method as described in any of clauses E01-E20, wherein the plurality of GPS location data points are captured during a traversal of the interior of the plurality of rooms, and wherein determining the one or more GPS location data points for the one or more locations on the floor plan corresponding to the at least one exterior wall of the building comprises extending one or more of the captured GPS data points from the interior of the plurality of rooms to the at least one exterior wall of the building.

[0274] E22. A computer-implemented method as described in any of clauses E01-E21, wherein the multiple types of additional information include two or more of an image of the interior of the building, or a video of the interior of the building, or a text description of the building, or a street-level view of the exterior of the building, or an overview view of the exterior of the building.

[0275] E23. The computer-implemented method of any one of claims E01-E22, wherein determining the one or more levels of detail for displaying the interior portion of the building on the map for each of the one or more stories of the building comprises determining a size on the map at which the visual representation of the floor plan is to be displayed based on at least one of a determined location on the map or a zoom level of the map, and selecting multiple levels of detail to display for the interior portion of the building based at least in part on the determined size.

[0276] E24. A computer-implemented method as described in clause E23, wherein the multiple levels of detail include two or more interior walls of the floor plan, or one or more types of appliances at indicated locations within the interior walls, or one or more types of fixtures at indicated locations on or within the interior walls, or one or more types of built-in structural elements at indicated locations on or within the interior walls, or one or more types of mounting elements at indicated locations on or within the interior walls, or one or more types of furniture located at indicated locations within the interior walls, wherein presenting the map with the visual representation includes: overlaying the visual representation of the floor plan on the map at the determined location, and prior to the overlaying, fitting an exterior boundary of the building on the floor plan to an existing visual representation of the building on the map, and wherein the overlaid visual representation includes a visual indication for an interior portion of the building having the multiple levels of detail.

[0277] E25. A computer-implemented method as described in any of clauses E01-E24, wherein the building has a plurality of stories, and wherein the method further comprises at least one of the following:

[0278] receiving one or more user interactions with the presented map after presenting the map with a visual representation of the floor plan, the visual representation including room shapes for at least one room on each of the plurality of stories, and in response to the one or more user interactions, updating the presented map to display an updated visual representation of the floor plan, the updated visual representation including only information for a single story from the plurality of stories; or

[0279] After presenting a map with a visual representation of the floor plan, receiving one or more user interactions with the presented map, the visual representation including a room shape for at least one room on a selected one of the plurality of stories, and in response to the one or more user interactions, updating the presented map to display an updated visual representation of the floor plan including additional room shapes for at least one additional room on an additional story of the plurality of stories, the additional room shapes being different from the additional story of the plurality of stories. Selecting a story.

[0280] E26. A computer-implemented method as described in any of clauses E24-E25, wherein the one or more user interactions include changing a zoom level of the map, and performing an update of the presented map based at least in part on the changed zoom level of the map.

[0281] E27. A computer-implemented method as described in clause E26, wherein determining the one or more levels of detail for displaying the interior portion of the building on the map for each of the one or more stories of the building comprises determining a size on the map at which the visual representation of the floor plan is to be displayed based on at least one of a determined location on the map or a zoom level of the map, and selecting multiple levels of detail to display for the interior portion of the building based at least in part on the determined size.

[0282] E28. A computer-implemented method as described in clause E27, wherein the multiple levels of detail include two or more interior walls of the floor plan, or one or more types of appliances at indicated locations within the interior walls, or one or more types of fixtures at indicated locations on or within the interior walls, or one or more types of built-in structural elements at indicated locations on or within the interior walls, or one or more types of mounting elements at indicated locations on or within the interior walls. Or one or more types of furniture located at indicated locations within the interior walls.

[0283] E29. A computer-implemented method as described in any of clauses E01-E28, wherein the building has a plurality of stories, and wherein the method further comprises at least one of the following:

[0284] receiving one or more user interactions with the presented map after presenting the map with a visual representation of the floor plan, the visual representation including room shapes for at least one room on each of the plurality of stories, and in response to the one or more user interactions, updating the presented map to display an updated visual representation of the floor plan, the updated visual representation including only information for a single story from the plurality of stories; or

[0285] After presenting a map with a visual representation of the floor plan, receiving one or more user interactions with the presented map, the visual representation including a room shape for at least one room on a selected one of the plurality of stories, and in response to the one or more user interactions, updating the presented map to display an updated visual representation of the floor plan including additional room shapes for at least one additional room on an additional story of the plurality of stories, the additional room shapes being different from the additional story of the plurality of stories. Selecting a story.

[0286] E30. A computer-implemented method as described in any of clauses E01-E29, wherein the one or more user interactions include changing a zoom level of the map, and performing an update of the presented map based at least in part on the changed zoom level of the map.

[0287] E31. A computer-implemented method as described in any of clauses E01-E30, wherein the absolute position data points include GPS (Global Positioning System) data points captured during a traversal of the interior of the one or more rooms, wherein the one or more locations on the at least partial floor plan correspond to one or more exterior walls of the building, and wherein determining the one or more absolute position data points for the one or more locations on the at least partial floor plan includes extending one or more of the one or more locations on the at least partial floor plan. The captured GPS data points point from the interior of the one or more rooms to the one or more exterior walls of the building.

[0288] E32. A computer-implemented method as described in any of clauses E01-E31, wherein the automated operation further comprises:

[0289] obtaining information about one or more external objects on a property including a building and external to the building;

[0290] And wherein presenting the map using the visual representation of the at least partial floor plan includes: for each of the one or more external objects, displaying an additional visual representation of the external object at an additional location on the map corresponding to the location of the external object on the property.

[0291] E33. A computer-implemented method as described in any of clauses E01-E32, wherein the automated operation further comprises:

[0292] obtaining, by the one or more computing devices, information about one or more external objects on a property including the building and external to the building;

[0293] And wherein presenting the map with the visual representation of the floor plan includes: for each of the one or more external objects, displaying an additional visual representation of the external object at an additional location on the map corresponding to the location of the external object on the property.

[0294] E34. The computer-implemented method of any one of claims E01-E33, wherein each of the external features is an object on the property outside the building, and wherein determining a location on the property for the external feature comprises at least one of analyzing visual data of an image that includes the external feature and has associated absolute location data or retrieving information about the external feature from one or more public data sources.

[0295] E35. A computer-implemented method as described in any of clauses E01-E34, wherein the at least partial floor plan includes a plurality of room shapes of two or more of the plurality of rooms positioned relative to each other, wherein the absolute position data points include GPS (Global Positioning System) data points captured during a traversal of the interior of the one or more rooms, wherein the one or more locations on the at least partial floor plan correspond to one or more exterior walls of the building. wherein determining the one or more absolute position data points of the one or more locations on the at least partial floor plan includes extending one or more of the captured GPS data points from the interior of the one or more rooms to the one or more exterior walls of the building, and wherein presenting the map includes presenting in a graphical user interface a visual representation of the at least partial floor plan overlaid on the map and a visual indication of the interior walls of the two or more rooms, and further includes presenting on the map one or more areas separate from the property.

[0296] E36. A computer-implemented method comprising a plurality of steps of performing automated operations, said automated operations implementing techniques substantially as disclosed herein.

[0297] F01. A non-transitory computer-readable medium having stored executable software instructions and / or other storage contents, wherein the stored executable software instructions and / or other storage contents enable one or more computing systems to perform automatic operations implementing the method of any of clauses E01-E36.

[0298] F02. A non-transitory computer-readable medium having stored executable software instructions and / or other stored content that causes one or more computing systems to perform automated operations that implement the techniques substantially as disclosed herein.

[0299] G01. One or more computing systems comprising one or more hardware processors and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations implementing the method of any of clauses E01-E36.

[0300] G02. One or more computing systems comprising one or more hardware processors and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations that implement techniques substantially as disclosed herein.

[0301] H01. A computer program adapted to perform the method of any of clauses E01-E36 when said computer program is run on a computer.

[0302] Here, the aspects of the present disclosure are described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure.It should be understood that each frame of the flowchart and / or block diagram and the combination of the frames in the flowchart and / or block diagram can be implemented by computer-readable program instructions.It will be further understood that in some implementations, the functions provided by the above routines can be provided in an alternative manner, such as splitting between more routines, or merging into fewer routines.Similarly, in some implementations, the routines shown can provide more or less functions than described, such as when other routines shown lack or include such functions respectively, or when the amount of functions provided changes.In addition, although various operations can be shown as being performed in a specific manner (e.g., serial or parallel, or synchronous or asynchronous) and / or in a specific order, in other implementations, operations can be performed in other orders and other ways.Any data structure discussed above can also be constructed in different ways, such as by dividing a single data structure into multiple data structures and / or by merging multiple data structures into a single data structure. Similarly, in some implementations, illustrated data structures may store more or less information than depicted, such as when other illustrated data structures lack or include such information, respectively, or when the amount or type of information stored varies.

[0303] It is understood from the above that, although specific embodiments are described herein for the purpose of illustration, various modifications may be made without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited except by the corresponding claims and the elements cited by those claims. In addition, although some aspects of the present invention may be presented in the form of certain claims at certain times, the inventors have considered various aspects of the present invention in the form of any available claims. For example, although only some aspects of the present invention may be described as being embodied in a computer-readable medium at a particular time, other aspects may also be embodied in the same manner.

Claims

1. A non-transitory computer-readable medium having stored content, the stored content causing one or more computing devices to perform an automated operation, the automated operation comprising at least: obtaining, by the one or more computing devices, a least partial floor plan of the building based at least in part on visual data of the plurality of images after obtaining a plurality of images and a plurality of GPS (Global Positioning System) location data points in a plurality of rooms of the building, wherein, for each of the plurality of rooms, the plurality of images comprises: images of the plurality of images acquired by a camera device at a first acquisition location in the room and having visual coverage of at least some walls of the room, wherein, for each of the plurality of rooms, the plurality of GPS location data points comprises: a plurality of GPS location data points of the plurality of GPS location data points captured by a mobile capture device at a plurality of second capture locations in the room and within a defined amount of time of acquisition of the images in the room, wherein the mobile capture device moves independently of the camera device, wherein the at least partial floor plan comprises determined room shapes of the plurality of rooms positioned relative to each other, each room shape for a room comprising a representation of at least some walls of the room, and wherein the at least partial floor plan comprises: information regarding a location on the at least partial floor plan of the first acquisition location using a coordinate system local to the at least partial floor plan; determining, by the one or more computing devices, and for each of the plurality of rooms, a representative GPS location data point associated with the first acquisition location in the room by using the plurality of GPS location data points captured at the plurality of second capture locations in the room; Determining, by the one or more computing devices, one or more GPS location data points for one or more locations on the at least partial plan view, comprising: for local coordinates of the one or more locations in a coordinate system local to the at least partial plan view, converting the local coordinates to the one or more GPS location data points; and A visual representation of the at least partial floor plan for the building is provided, by the one or more computing devices, for inclusion on a map at one or more map locations corresponding to the one or more GPS location data points determined for the one or more locations.

2. The non-transitory computer-readable medium of claim 1, wherein: The stored content includes software instructions, which, when executed by the one or more computing devices, cause the one or more computing devices to perform further automatic operations, the further automatic operations including: determining, by the one or more computing devices, a global transformation, the global transformation mapping data in a first data set to data in a second data set, the first data set including positions in the at least partial plan view of the first acquisition location using a coordinate system local to the at least partial plan view, and the second data set including the representative GPS location data points associated with the first acquisition location, and wherein converting local coordinates in the coordinate system local to the at least partial plan view to the one or more GPS location data points includes: using the global transformation.

3. The non-transitory computer-readable medium of claim 1, wherein: The stored content includes software instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform further automated operations, the further automated operations comprising: determining, by the one or more computing devices, a global transformation that maps data in a first data set to data in a second data set, the first data set comprising locations in the at least partial plan view of the first acquisition location using a coordinate system local to the at least partial plan view, and the second data set comprising the representative GPS location data points associated with the first acquisition location, and wherein converting local coordinates in the coordinate system local to the at least partial plan view to the one or more GPS location data points comprises: determining, by the one or more computing devices, a revised representative GPS location data point for each of at least some of the first acquisition locations based on the global transformation; and Extending, by means of the one or more computing devices, GPS location data from each of one or more of at least some of the first acquisition locations to one or more walls of the room shape of the room in which the first acquisition locations are located, wherein the walls to which the GPS location data is extended include one or more exterior walls of the building, the one or more exterior walls having the one or more locations on the at least partial floor plan.

4. The non-transitory computer readable medium of claim 1, wherein: The visual representation of the at least partial floor plan is at least one of a two-dimensional floor plan model or a three-dimensional floor plan model, and wherein determining the one or more GPS location data points for the one or more locations on the at least partial floor plan includes determining the one or more GPS location data points for one or more locations corresponding to one or more exterior walls of the building.

5. The non-transitory computer readable medium of claim 1, wherein: The mobile capture device capturing the multiple GPS location data points at multiple second locations includes: capturing additional GPS location data points in the one room at additional times, and wherein the automatic operation also includes: selecting the multiple GPS location data points for determining the representative GPS location data of the first acquisition location in the one room based at least in part on the multiple GPS location data points captured within a limited amount of time for acquiring the image in the one room.

6. The non-transitory computer readable medium of claim 5, wherein: The camera device lacks any GPS receiver, wherein the automatic operation further comprises: prior to selecting the plurality of GPS location data points, performing a smoothing operation on the additional GPS location data points, the smoothing operation comprising removing one or more of the additional GPS location data points, each of the one or more additional GPS location data points being determined to be an outlier, and wherein determining the representative GPS location data point associated with the first acquisition location in the one room comprises: retrieving, by the one or more computing devices, an acquisition time of the image acquired at the first acquisition location in the one room; selecting, by the one or more computing devices, a plurality of GPS location data points associated with the first acquisition location in the one room based at least in part on a plurality of GPS location data points having a capture time within a defined time window around the retrieved acquisition time, the defined time window being based on the defined amount of time; and The representative GPS location data point associated with the first acquisition location in the one room is generated, by the one or more computing devices, based at least in part on combining location data of a plurality of GPS location data points of a plurality of second capture locations in the one room.

7. The non-transitory computer-readable medium of claim 1, wherein: Determining the representative GPS location data point associated with the first acquisition location in one of the multiple rooms includes: selecting, by the one or more computing devices, one of the multiple GPS location data points captured in the one room to be used as the representative GPS location data point associated with the first acquisition location in the one room, the selection being based at least in part on the acquisition time of the image in the one room and the capture time of the one GPS location data point.

8. The non-transitory computer readable medium of claim 1, wherein: Obtaining the at least partial floor plan includes: for each of the multiple rooms, determining a room shape of the room, the room shape including at least some walls of the room and the position of the first acquisition location in the room within the room shape, and also includes: positioning the room shapes of the multiple rooms relative to each other.

9. The non-transitory computer readable medium of claim 1, wherein: Providing the visual representation of the at least portion of the floor plan for the building includes: superimposing the visual representation of the at least portion of the floor plan on the map, by the one or more computing devices, at the one or more map locations corresponding to the one or more GPS location data points determined for the one or more locations, the map covering an area larger than the real estate on which the building is located, and also includes: sending the map with the overlaid visual representation to a client device for display.

10. The non-transitory computer readable medium of claim 1, wherein: The automatic operation also includes: acquiring, by the one or more computing devices, a plurality of additional GPS location data points of the exterior of the building, the plurality of additional GPS location data points being captured by the mobile capture device on the real estate on which the building is located; and determining, by the one or more computing devices, at least one of the plurality of additional GPS location data points to represent an exterior doorway of the building, the determination being based on analyzing additional visual data including one or more additional acquired images of the exterior doorway to determine when the mobile capture device is proximate to the exterior doorway, and Wherein, determining the one or more GPS location data points for the one or more locations on the at least partial floor plan, and at least one of the visual representations of locating the at least partial floor plan for the building on the map is also based in part on at least one additional GPS location data point determined to represent the exterior doorway.

11. The non-transitory computer readable medium of claim 1, wherein: The automatic operation also includes: acquiring, by the one or more computing devices, a plurality of additional GPS location data points exterior to the building, the plurality of additional GPS location data points being captured by the mobile capture device on real estate on which the building is located, wherein the plurality of additional GPS location data points are captured during a traversal of an exterior boundary of the building; and determining, by the one or more computing devices, at least one of a shape or a position of the exterior boundary of the building based at least in part on the plurality of additional GPS location data points, and Wherein, determining the one or more GPS location data points for the one or more locations on the at least partial floor plan, and locating on the map at least one of the visual representations of the at least partial floor plan for the building is also based in part on at least one of the determined shape or the position of the external boundary.

12. The non-transitory computer readable medium of claim 1, wherein: The automatic operation also includes: obtaining, by the one or more computing devices, a plurality of additional GPS location data points of the exterior of the building, the plurality of additional GPS location data points being captured by the mobile capture device on real estate on which the building is located, wherein the plurality of additional GPS location data points are captured during a traversal from an exterior walkway to the building and correspond to a substantially straight line; and determining, by the one or more computing devices, a location of the exterior walkway based at least in part on the plurality of additional GPS location data points, and Wherein, determining the one or more GPS location data points for the one or more locations on the at least partial floor plan, and locating at least one of the visual representations of the at least partial floor plan for the building on the map are also based in part on the determined location of the exterior walkway.

13. The non-transitory computer readable medium of claim 1, wherein: Prior to obtaining the plurality of GPS location data points in the plurality of rooms, the automatic operation further comprises: providing, by the one or more computing devices, instructions related to a movement pattern of the mobile capture device outside the building to enable an increase in accuracy of the plurality of GPS location data points in the plurality of rooms during subsequent capture of the plurality of GPS location data points in the plurality of rooms; and A plurality of additional GPS location data points captured by the mobile capture device and corresponding to the movement pattern of the mobile capture device outside the building are obtained, via the one or more computing devices.

14. A computer-implemented method comprising: Obtaining, by one or more computing devices, a plurality of images and a plurality of GPS (Global Positioning System) location data points in a plurality of rooms of a building, for each of the plurality of rooms, comprising: obtaining one or more of the plurality of images and a plurality of the plurality of GPS location data points, the plurality of images being acquired by a camera device at a first acquisition location in the room and having visual coverage of a wall of the room, a plurality of the plurality of GPS location data points being simultaneously captured by a mobile capture device at a plurality of second capture locations in the room, wherein the mobile capture device moves independently of the camera device, obtaining, by the one or more computing devices and based on the visual data of the plurality of images, a floor plan of a building and information about the position of the first acquisition location on the floor plan using a coordinate system local to the floor plan, including: for each of the plurality of rooms, determining a room shape of the room, the room shape of the room including at least some walls of the room and the position of the first acquisition location in the room within the room shape, and further including positioning the room shapes of the plurality of rooms relative to each other; determining, by the one or more computing devices, and for each of the plurality of rooms, a representative GPS location data point associated with the first acquisition location in the room, comprising: identifying two or more candidate GPS location data points from the plurality of GPS location data points captured in the room using capture times of the two or more candidate GPS location data points, and generating the representative GPS location data point for the first acquisition location in the room using the two or more candidate GPS location data points; determining, by the one or more computing devices, a global transformation that maps data in a first data set to data in a second data set, the first data set comprising locations in the plan view of the first acquisition location using a coordinate system local to the plan view, and the second data set comprising the representative GPS location data points associated with the first acquisition location; Determining, by the one or more computing devices, one or more GPS location data points of one or more locations on the floor plan corresponding to one or more locations on the exterior walls of the building, comprising: for the one or more locations on the one or more exterior walls, using the global transformation, converting local coordinates of the one or more locations in a coordinate system local to the floor plan into the one or more GPS location data points; A visual representation of the floor plan for the building is provided, via the one or more computing devices, the visual representation being superimposed on a map at map locations corresponding to the determined one or more GPS location data points for one or more locations on the one or more exterior walls of the building.

15. The computer-implemented method of claim 14, wherein: The two or more candidate GPS location data points associated with the first acquisition location in one of the multiple rooms are captured at multiple times and at multiple capture locations, and the two or more candidate GPS location data points are a subset of the multiple GPS location data points acquired in the one room after performing a smoothing operation to remove outlier data points, wherein the camera device lacks any GPS receiver, and wherein determining the representative GPS location data point associated with the first acquisition location in the one room comprises: retrieving, by the one or more computing devices, a capture time of one of the images captured at the first capture location in the one room; determining, by the one or more computing devices, two or more candidate GPS location data points associated with the first acquisition location in the one room based at least in part on the two or more candidate GPS location data points having capture times within a defined time window around the retrieved acquisition time; and The representative GPS location data point associated with the first acquisition location in the one room is generated, by the one or more computing devices, based at least in part on average location data of multiple capture locations of the two or more candidate GPS location data points associated with the first acquisition location in the one room.

16. The computer-implemented method of claim 14, wherein: Determining the one or more GPS location data points for one or more locations on the floor plan corresponding to one or more locations on the exterior walls of the building includes: determining, by the one or more computing devices, a revised representative GPS location data point for each of at least some of the first acquisition locations based on the global transformation; and extending the GPS location data from each of the one or more of the first acquisition locations to one or more walls of the room shape of the room in which the first acquisition location is located, wherein the walls to which the GPS location data extends include the one or more exterior walls.

17. The computer-implemented method of claim 14, wherein: Determining the global transformation includes at least one of the following: Determining, by the one or more computing devices, a rigid transformation between the first data set and the second data set using a Kabsch algorithm; or determining, by the one or more computing devices, a rigid transformation between the first data set and the second data set using a quaternion estimation algorithm; or Determining, by the one or more computing devices, a rigid transformation between the first data set and the second data set using an algorithm capable of solving a Wahba problem; or determining, by the one or more computing devices, the global transformation using a point set registration technique; or The global transformation is determined, by the one or more computing devices, using a trained machine learning model, the trained machine learning model taking at least the first data set and the second data set as input and producing the global transformation as output.

18. A system comprising: one or more hardware processors of one or more computing devices; as well as One or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations, the automated operations comprising at least: Obtaining a plurality of images and a plurality of absolute location data points in one or more rooms of a building, for each of the one or more rooms, comprising: obtaining an image of the plurality of images, the image being acquired by a camera device at each of one or more first acquisition locations in the room and having visual coverage of at least some walls of the room; obtaining a plurality of absolute location data points of the plurality of absolute location data points, captured by a mobile capture device at a plurality of second capture locations in the room, the mobile capture device being movable independently of the camera device; obtaining, based at least in part on the visual data of the plurality of images and for each of the one or more rooms, a room shape of the room and information about a position in the room shape of each of the one or more first acquisition locations in the room using a coordinate system local to the room shape, the room shape of the room including representations of at least some walls of the room; for each of the one or more rooms, and by using the plurality of absolute location data points captured at the plurality of second capture locations in the room, determining a representative absolute location data point associated with each of the one or more first capture locations in the room; determining a global transformation that maps data in a first data set to data in a second data set, the first data set comprising, for each of the one or more rooms, locations of the one or more first acquisition locations in the room using a coordinate system local to the room shape of the room, and the second data set comprising representative GPS location data points associated with the first acquisition locations; For each of at least one room shape of the one or more rooms, determining one or more absolute location data points for one or more locations on the room shape, comprising: using the global transformation, for the local coordinates of the one or more locations, converting the local coordinates of the one or more locations in a local coordinate system of the room shape to one or more absolute location data points; Using the determined one or more absolute location data points for each of the at least one room shape, a visual representation of the at least one room shape is provided.

19. The system of claim 18, wherein: The absolute location data points are GPS (Global Positioning System) data points, and wherein the one or more locations on each of the at least one room shape include one or more locations corresponding to one or more exterior walls of the building.

20. The system of claim 18, wherein: The plurality of images and the plurality of absolute location data points include images and absolute location data points from a plurality of rooms of the building, and wherein obtaining a room shape for each of the one or more rooms includes: for each of the plurality of rooms, determining a room shape of the room, the room shape comprising at least some of the walls of the room and a position of a first acquisition location in the room within the room shape; positioning the room shapes of the plurality of rooms relative to each other to form at least a partial floor plan of the building; and Wherein, providing the visual representation of at least one room shape of the building includes: superimposing the visual representation of the at least partial floor plan on a map at the one or more map locations corresponding to the one or more absolute location data points of each of the at least one room shape determined, the map covering an area larger than the real estate in which the building is located; and also includes transmitting the map with the superimposed visual representation to a client device for display.

21. A computer-implemented method comprising: obtaining, by one or more computing devices, a floor plan for a building having a plurality of rooms and information about a plurality of GPS (Global Positioning System) location data points captured during a traversal of interiors of the plurality of rooms, the floor plan including room shapes of the plurality of rooms positioned relative to one another; determining, by the one or more computing devices, and based at least in part on the plurality of GPS location data points, one or more GPS location data points for one or more locations on the floor plan corresponding to at least one exterior wall of the building, including: extending one or more of the captured GPS data points from the interior of the plurality of rooms to the at least one exterior wall of the building; determining, by the one or more computing devices and based at least in part on the determined one or more GPS location data points for the one or more locations on the floor plan, a location corresponding to the floor plan on a map of an area including the building; determining, by the one or more computing devices and based at least in part on the determined locations on the map, one or more levels of detail for an interior portion of the building to display each of the one or more floors of the building on the map; and A map having a visual representation of the floor plan is presented, via the one or more computing devices and as part of a graphical user interface (GUI), the floor plan being superimposed on the map at the determined location and including a visual indication of an interior portion of the building having the determined one or more levels of detail and including at least interior walls of the plurality of rooms.

22. A system comprising: one or more hardware processors of one or more computing devices; as well as One or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations, the automated operations comprising at least: For a building having a plurality of rooms, obtaining at least a partial floor plan of the building, the at least partial floor plan comprising room shapes of one or more rooms of the plurality of rooms and information about a plurality of absolute location data points captured in the one or more rooms; determining one or more absolute location data points for one or more locations on the at least partial plan view based at least in part on the plurality of absolute location data points; determining, on a map of an area including the building, a location corresponding to the at least partial plan view based at least in part on the determined one or more absolute location data points for the one or more locations on the at least partial plan view; determining one or more levels of detail for an interior portion of the building to display each of the one or more floors of the building on the map; and A visual representation of the at least partial floor plan is presented to the map, the visual representation being displayed at the determined location on the map and including data corresponding to the determined one or more levels of detail of the interior portion of the building.

23. A non-transitory computer-readable medium having stored content, the stored content causing one or more computing devices to perform an automated operation, the automated operation comprising at least: obtaining, by one or more computing devices, a floor plan for a building having a plurality of rooms and information about a plurality of GPS (Global Positioning System) location data points captured during a traversal of the plurality of rooms, the floor plan including room shapes of the plurality of rooms positioned relative to each other; determining, by the one or more computing devices and based at least in part on the plurality of GPS location data points, one or more GPS location data points for one or more locations on the floor plan corresponding to at least one exterior wall of the building; determining, by the one or more computing devices and based at least in part on the determined one or more GPS location data points for the one or more locations on the floor plan, a location corresponding to the floor plan on a map of an area including the building; retrieving, by the one or more computing devices, additional information about a plurality of types of the buildings; as well as The map is presented via the one or more computing devices and as part of a graphical user interface (GUI), the map having a visual representation of the floor plan at the determined location superimposed in a first portion of the GUI and additional portions of the GUI, each of the additional portions of the GUI including at least some additional information for a corresponding type of the plurality of types.

24. A computer-implemented method comprising: Obtaining, by one or more computing devices, and for a building on real estate, at least a partial floor plan for the building, the at least partial floor plan comprising: room shapes of one or more rooms of the building; and first information regarding a plurality of absolute location data points captured in the one or more rooms; and second information regarding an exterior of the building and one or more exterior features on the real estate; determining, by the one or more computing devices and based at least in part on the plurality of absolute location data points, one or more absolute location data points for one or more locations on the at least partial plan view; determining, by the one or more computing devices and based at least in part on the determined one or more absolute location data points for the one or more locations on the at least partial floor plan, a location corresponding to the at least partial floor plan on a map that includes the real estate; determining, by the one or more computing devices, and for each of the one or more exterior features, a location of the exterior feature on the real estate; and A visual representation of the at least a portion of the floor plan shown at the determined location on the map is presented to the map, by the one or more computing devices, and also, for each of the one or more exterior features, an additional visual representation of the exterior feature is presented at the determined location of the exterior feature on the real estate.

Images