Automatically determine image acquisition locations within building interiors using determined room shapes

By automatically determining the image acquisition location of a building using a computing device and analyzing the image content using a neural network to generate a building floor plan, this technology solves the problem of capturing and representing internal building information in existing technologies, and achieves efficient and accurate image acquisition location determination and navigation support.

CN114529694BActive Publication Date: 2024-11-19MFTB CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111395691.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-03-15
Filing Date
2021-11-23
Publication Date
2024-11-19
Estimated Expiration
2041-11-23

AI Technical Summary

Technical Problem

Existing technologies struggle to effectively capture, represent, and use information about the interior of buildings, especially without physical access, and building floor plans are difficult to construct, maintain, accurately scale, and visualize in certain situations.

Method used

The computer automatically determines the acquisition location of the target image in the building, generates a floor plan of the building, analyzes the image content using a neural network, estimates the room shape, and matches candidate shapes to locate the image position, providing acquisition location information.

Benefits of technology

It enables rapid and high-precision determination of image acquisition location without the use of depth sensors, supports building navigation and information presentation, and improves the accuracy and efficiency of information acquisition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114529694B_ABST
    Figure CN114529694B_ABST
Patent Text Reader

Abstract

Techniques are described for a computing device to perform automated operations to determine the acquisition location of an image, such as the automatically determined shape of a room within a building, and techniques for using the determined image acquisition location information in a further automated manner. The image may be a panoramic image or another type (e.g., a linear stereoscopic image), acquired at an acquisition location within a multi-room building, and the acquisition location used to determine such an image may be at least one location on a floor plan of the building, and optionally the orientation / direction of at least a portion of the image. Furthermore, automated image acquisition location determination can be further performed without having or using information from any depth sensor or other distance measuring device regarding the distance from the image acquisition location to walls or other objects in the surrounding building.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The following disclosure generally relates to techniques for automatically determining an acquisition location for an image within a building interior based on an automatically determined room shape, and subsequently using the determined acquisition location information in one or more ways, such as locating an image of an interior of a room in the building on a floor plan of the building and using the image location to improve navigation of the building. Background Art

[0002] In various fields and environments, such as architectural analysis, property inspections, real estate acquisition and development, renovation and improvement services, general contracting, and other environments, it may be desirable to view information about the interior of a house, office, or other building without having to physically travel to and enter the building, including determining actual as-built information about the building rather than design information from before the building was built. However, it may be difficult to effectively capture, represent, and use such building interior information, including displaying visual information captured inside the building to a user at a remote location (e.g., enabling the user to fully understand the layout and other details of the interior, including controlling the display in a manner selected by the user). In addition, although a floor plan of a building can provide some information about the layout and other details of the interior of the building, the use of such a floor plan has some disadvantages in some cases, including that the floor plan may be difficult to construct and maintain, difficult to accurately scale and populate with information about the interior of a room, difficult to visualize and use, etc. Summary of the invention

[0003] The present disclosure provides a system, comprising one or more hardware processors of one or more computing devices; and one or more memories having stored instructions, wherein when the stored instructions are executed by at least one of the one or more hardware processors, at least one of the one or more computing devices performs automatic operations, wherein the automatic operations at least include: for a target image acquired in a building having a plurality of rooms, determining a capture location of the target image in the building, including: for each of one or more of the plurality of rooms, acquiring a determined room shape of the room; generating an estimated room shape of the target image based at least in part on analyzing visual content of the target image; selecting a plurality of candidate shape matches for the estimated room shape based at least in part on a match degree determined for a selected candidate shape match; one of the combinations, wherein the plurality of candidate shape matches each include an arrangement of one of the acquired determined room shapes of one of the plurality of rooms relative to the estimated room shape and have a determined degree of match based at least in part on a fit of the estimated room shape to the arrangement of one of the acquired determined room shapes; locating the acquisition position of the target image based on the arrangement of the selected one of the candidate shape matches and by using information about the target image within the estimated room shape within the room in which the acquired determined room shape is used for the selected one of the candidate shape matches; and providing information about the determined acquisition position of the target image in the room of the building in which the acquired determined room shape is used for the selected one of the candidate shape matches.

[0004] Another aspect of the present disclosure provides a computer-implemented method, comprising: receiving, by one or more computing systems, a target image acquired in a building having a plurality of rooms, wherein the target image is a panoramic image having a visual overlay of a plurality of surrounding walls;

[0005] Determining, by the one or more computing systems, a capture location of the target image on a floor plan of the building, comprising: obtaining, by the one or more computing systems, determined three-dimensional (“3D”) room shapes of the plurality of rooms; generating, by the one or more computing systems and based at least in part on analyzing visual content of the target image, an estimated 3D room shape visible in the target image and an indicated location of the target image within the estimated 3D room shape; selecting, by the one or more computing systems, one of a plurality of candidate shape matches for the estimated 3D room shape based at least in part on a degree of match determined for the selected one of the candidate shape matches, wherein the plurality of candidate shape matches each include an arrangement of one of the determined 3D room shapes relative to the estimated 3D room shape and have a degree of match determined at least in part based on the one a determined degree of fit of a determined 3D room shape with the arrangement of the estimated 3D room shape, and wherein the one determined 3D room shape that the selected one candidate shape matches is an identified room of the plurality of rooms; locating, by the one or more computing systems, the indicated position regarding the arrangement within the estimated 3D room shape using the determined 3D room shape relative to the arrangement that the selected one candidate shape matches for the estimated 3D room shape, and identifying, on the floor plan, the acquisition position of the target image; and providing, by the one or more computing systems, information regarding the determined acquisition position of the target image in the identified room to enable display of the determined acquisition position on the floor plan of the building.

[0006] The present disclosure also provides a non-transitory computer-readable medium having storage content, the storage content causing one or more computing systems to perform automatic operations, the automatic operations comprising at least: receiving, by the one or more computing systems, a target image and a plurality of additional images acquired in a building having a plurality of rooms, wherein each of the target image and the plurality of additional images is a panoramic image with a visual overlay of a plurality of surrounding walls; determining, by the one or more computing systems, a capture location of the target image in the building, comprising: acquiring, by the one or more computing systems, a three-dimensional (“3D”) room shape determined by the plurality of additional images, wherein for an associated one of the plurality of additional images, each determined 3D room shape at least identifies walls and corners visible in surrounding rooms; generating, by the one or more computing systems and based at least in part on analyzing the visual content of the target image, an estimated 3D room shape and an indicated location of the target image within the estimated 3D room shape, the estimated 3D room shape including at least walls and corners visible in the target image; determining, by the one or more computing systems, based at least in part on selecting a selected one of the plurality of additional images; selecting, by the one or more computing systems and in the surrounding room, for the additional images associated with the one determined 3D room shapes of the selected one candidate shape matches, one of a plurality of candidate shape matches for the estimated 3D room shape, wherein the plurality of candidate shape matches each include an arrangement of one of the determined 3D room shapes relative to the estimated 3D room shape and have a determined degree of match, the determined degree of match being based at least in part on a fit of the walls and corners of the estimated 3D room shape using the candidate shape matches with the arrangement of the walls and corners of the one determined 3D room shape of the candidate shape matches; locating, by the one or more computing systems and in the surrounding room, the indicated location within the estimated 3D room shape using the arrangement of the selected one candidate shape matches, and identifying the acquisition location of the target image within the building based on the located indicated location within the surrounding room; and providing, by the one or more computing systems, information regarding the acquisition location of the determined target image within the surrounding room to enable display of the determined acquisition location using information about the building. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1A-1B is a diagram depicting an exemplary building interior environment and computing system used in embodiments of the present disclosure, such as for performing automated operations to determine a capture location for an image within a building interior based on an automatically determined room shape, and subsequently using the determined capture location information in one or more ways.

[0008] Figure 2A-2W Examples of automated operations for generating and presenting information about a floor plan of a building based on images taken inside the building are shown, such as automatically determining and presenting the acquisition location of such images from analysis of image content.

[0009] Figure 3 is a block diagram of a computing system illustrating an embodiment of one or more systems suitable for performing at least some of the techniques described in this disclosure.

[0010] Figure 4 An example flow diagram of an image capture and analysis (ICA) system routine is shown in accordance with an embodiment of the present disclosure.

[0011] Figure 5A-Figure 5B An example flow chart of a mapping information generation manager (MIGM) system routine according to an embodiment of the present disclosure is shown.

[0012] Figure 6A-6B An example flow diagram of an Image Location Determination Manager (ILDM) system routine is shown in accordance with an embodiment of the present disclosure.

[0013] Figure 7 An example flow diagram of a building map viewer system routine is shown according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0014] The present disclosure describes techniques for using a computing device to perform automated operations related to determining a capture location for an image within an interior of a building based on an automatically determined shape of the rooms of the building, and for subsequently using the determined image capture location information in one or more other automated manners. The images to be analyzed may include panoramic images or other images (e.g., rectilinear stereo images) acquired at a capture location within an interior of a multi-room building (e.g., a house, office, etc.), collectively referred to herein as "target images," and the image capture location information determined for such target images includes at least a location on a floor plan of the building, and in some cases also includes an orientation or other direction information for at least a portion of the target image. In addition, in at least some such embodiments, the automatic determination of the image capture location is further performed without having or using information from any depth sensor or other distance measurement device regarding the distance from the capture location of the target image to walls or other objects in the surrounding building. The determined image acquisition location information for one or more target images acquired for a building may be further used in various ways in various embodiments, such as in conjunction with a corresponding building floor plan and / or other generated mapping-related information for the building, including for controlling navigation of a mobile device (e.g., an autonomous vehicle), for display or other presentation over a computer network on one or more client devices in a corresponding GUI (graphical user interface), etc. Additional details regarding the automatic determination and use of image acquisition location information are included below, and in at least some embodiments, some or all of the techniques described herein may be performed via automatic operation of an image location determination manager ("ILDM") system, as discussed further below.

[0015] As described above, the automatic operation of the ILDM system may include determining the acquisition position and optionally the orientation of a target image captured in a room of a house or other building (or in another defined area) by using automatically determined information about the shape of the room of the building (or a plurality of other defined shapes that are candidates for the acquisition position of the image). The combination of the acquisition position and orientation of the target image is sometimes referred to herein as the "acquisition position" or simply the "position" of the target image. For example, in at least some embodiments of automatically determining the acquisition position of the target image within the building, a building floor plan with associated room shape information for the rooms of the building may be used, and in at least some such cases, the 2D and / or 3D room shape of the room displayed on the floor plan (or other defined area) may have been automatically determined in advance (as further discussed below), while in other cases, some or all of the room shape of the room (or other shapes of the defined area) may be determined in other ways (e.g., automatically determined simultaneously when determining the acquisition position of the target image, determined in advance or simultaneously at least in part based on manual input of one or more users, etc.). The building floor plan with associated room shape information can have various forms in various embodiments, such as a 2D (two-dimensional) floor plan of the building (e.g., an orthographic top view or other overhead view of a schematic floor plan that does not include or display height information) and / or a 3D (three-dimensional) or 2.5D (two and a half-dimensional) floor plan model of the building that displays height information. In at least some embodiments, automatic determination of a capture location of a target image within a multi-room building can generally include: retrieving determined room shapes for some or all of the rooms of the building, and identifying one of those rooms whose determined room shape most closely matches the room shape of the target image, the room shape being estimated from the visual content of the target image and being placed (e.g., resized and rotated) to match the determined room shape of the identified room, including comparing aspects of the visual content of the target image with the determined room shape of the identified room to at least determine a location within the identified room to capture the target image.

[0016] As part of automated operations for identifying an estimated room shape of a room containing a target image based at least in part on the visual content of the target image, in at least some embodiments, the described techniques may include using one or more trained neural networks or other techniques to estimate the 3D room shape shown in the target image. As non-exclusive examples, such 3D room shape estimation may include one or more of the following: using a trained convolutional neural network or other analysis technique to take a target image as input and estimate 3D point clouds of walls and other surfaces of an enclosed room from the visual content of the target image, and / or estimating 3D walls and other planar surfaces of an enclosed room from the visual content of the target image; using a trained neural network or other analysis technique to take a target image as input and estimate wireframe structural lines of an enclosed room from the visual content of the target image (e.g., structural lines for showing one or more boundaries between walls, boundaries between walls and ceilings, boundaries between walls and floors, outlines of doorways and / or other interior room wall openings, outlines of windows, etc.); using a trained neural network or other analysis technique to detect wall structural elements in the visual content of the target image (e.g., windows and / or skylights; passages into and / or out of a room, such as at the top of a wall); doorways and other openings in walls, stairs, hallways, etc.; boundaries between adjacent walls; boundaries between a wall and a floor; boundaries between a wall and a ceiling; corners (or solid geometric vertices) where at least three surfaces or planes meet; etc.), and optionally detect other fixed structural elements (e.g., countertops, tubs, sinks, islands, fireplaces, etc.), and generate 3D bounding boxes for the detected elements, and optionally further determine object type labels for those elements (e.g., windows, doorways, etc.); determine room type labels for enclosed rooms (e.g., living rooms, bedrooms, bathrooms, kitchens, etc.) from the visual content of a target image using a trained neural network or other analysis technique; generate embedding vectors using a trained deep neural network or other analysis technique that encode information about elements and other features of an enclosed room, such as element type and location, room type and shape and size, etc. (e.g., in a concise format that is not easily discernible by a human reader); etc. In some embodiments, the acquisition metadata for the target image can be further used as part of determining an estimated room shape of the room containing the target image, such as by using data from an IMU (internal measurement unit) sensor on an acquisition camera or other associated device as part of performing SLAM (simultaneous localization and mapping) and / or SfM (structure from motion) and / or MVS (multi-view stereo) analysis, as further discussed below. The following includes additional details about automated operations that the ILDM system can perform in at least some embodiments for identifying an estimated room shape of the room containing the target image based at least in part on the visual content of the target image.

[0017] Once the estimated room shape of the target image of the enclosed room is automatically identified, it can be compared to other candidate room shapes (e.g., previously determined room shapes for some or all rooms of the building) to automatically determine a room shape from the candidates that best matches the estimated room shape of the target image, with rooms having such matching determined room shapes being referred to herein as "target" rooms within the building in which the acquisition location of the target image occurs. Determining one or more potentially matching candidate room shapes, including a particular arrangement (i.e., position and rotation) of the estimated room shape of the target image relative to the candidate room shapes, can include considering and evaluating a plurality of alternative shape matches, each alternative shape match having a different such arrangement of the estimated room shape of the target image relative to one or more candidate room shapes. For example, in at least some embodiments, one or more corners of the estimated room shape of the target image can be matched with one or more corners of the candidate room shapes successively or simultaneously (e.g., to match each corner of the estimated room shape of the target image with each corner of each candidate room shape, to match each pair of two corners of the estimated room shape of the target image with each pair of other two corners of each candidate room shape, etc.), together with using vanishing line angle information for the target image and the candidate room shape (or an existing image with a specified position within the candidate room shape) to determine wall alignment options for the two room shapes having matching corners, so that if consideration is given to For all alternatives of the estimated room shape of the target image and a particular candidate room shape, and there are M predicted room corners in the estimated room shape of the target image, and there are N room corners in the candidate room shape, and there are 2 vanishing line corners in each of the target image and the candidate room shape (or for an existing image with a specified position within the candidate room shape), there will be M*N*4 shape matching alternatives for the estimated room shape of the target image and the candidate room shape, each shape matching alternative providing a possible acquisition position and optionally an orientation for the target image according to the shape matching alternative, wherein such shape matching alternative is also considered for each candidate room shape.

[0018] In addition, in at least some embodiments, additional automatic operations can be performed to determine one or more preferred candidate room shapes to compare with the estimated room shape of the target image (e.g., by ranking some or all of the candidate room shapes and, in some embodiments, comparing the preferred candidates to the estimated room shape of the target image before or in place of other non-preferred candidates), such as by using one or more types of information to identify the preferred candidates, as follows: matching determined room type labels for the estimated room shape of the target image with room type labels for the candidate room shapes; using information about one or more floor plan rooms associated with one or more other images successively acquired in the building before and / or after the target image to identify the same or adjacent rooms in the floor plan as possible acquisition locations of the target image; pairwise comparing the visual content of the target image with the visual content of existing images having specified locations within the floor plan (e.g., by a trained neural network) to determine matching features in the paired images, the matching features enabling the location of the target image to be determined relative to the specified location of the paired existing images (e.g., having more than a first predetermined distance); by comparing the visual content of the entire target image with the visual content of the entire existing image having a specified position within the room shape of the floor plan in pairs (e.g., by a trained neural network) so as to determine similarities from the common visible areas in the paired images, so as to be able to determine the position of the target image relative to the specified position of the paired existing images (e.g., having a similarity exceeding a second defined matching threshold); determining the spatial relationship information of the target image, such as approximate 3D shape information of the surrounding environment, approximate 3D planar surfaces in the surrounding environment, camera travel path information in a closed room, etc., by using motion tracking information of the camera of the target image (e.g., using SLAM); determining, relative to the pairwise comparison of the visual content of the target image with the visual content of the existing image having a specified position within the room shape of the floor plan (e.g., by a trained neural network), multiple directions between the paired images to help determine the position of the target image relative to the specified position of the paired existing images (e.g., having a similarity exceeding a third defined matching threshold in multiple directions), by using the motion tracking information of the camera of the target image (e.g., using SLAM);

[0019] To select between the estimated room shape for the target image and at least one candidate room shape shape matching alternative, for each shape matching alternative, one or more matching scores may be considered and combined (if more than one is considered), such as by weighting or a trained machine learning model for combination, non-exclusive examples of such matching scores include the following: scoring the distance between the estimated room shape for the target image and the location of each corner of the room shape of the candidate for the shape matching alternative, using a particular location of the estimated room shape of the target image for the shape matching alternative, and after re-projecting the candidate room shape into the target image space, and identifying a better match with a smaller distance ; scoring the distance between the wireframe structure lines of the estimated room shape of the target image and the wireframe structure lines of the candidate room shape of the shape matching candidate, using a specific position of the estimated room shape of the target image for the shape matching candidate, and identifying a better match using a smaller distance after reprojecting the wireframe structure lines of the candidate room shape into the target image space; scoring the distance between the position of each object 3D bounding box of the estimated room shape of the target image and the position of the candidate room shape of the shape matching candidate, using a specific arrangement of the estimated room shape of the target image for the shape matching candidate and after reprojecting the object 3D bounding box of the candidate room shape into the target image space, such as by using a cross span Joint distance measurement, and identifying better matches with smaller distances; scoring the distance between the location of the structural walls of the estimated room shape of the target image and the location of the structural walls of the candidate room shape of the shape matching alternative, using the specific arrangement of the estimated room shape of the target image of the shape matching alternative, and after reprojecting the structural walls of the candidate room shape into the target image space, and identifying better matches with smaller distances; scoring the difference between two determinations of the direction / angle between the acquisition location of the target image of the shape matching alternative and the specified location in the existing image of the candidate room shape, for example by using the determined location and direction of the acquisition location of the shape matching alternative relative to the specified location in the existing image as the first a direction / angle, and using a comparison of the visual content of the target image with an existing image (e.g., using SfM analysis; using the results of a trained convolutional neural network; etc.) to provide a second direction / angle, and identifying a better match with a smaller difference; using the shape matching alternative candidate room shape to score a difference between the total visual content of the target image and another image in the room (e.g., additional images acquired in the room at the same time as the target image, such as within a defined time period of minutes and / or hours; existing images previously acquired in the room and having a specified position in the room, etc., and as determined by a trained convolutional neural network or in another manner), and identifying a better match with a smaller difference;Scoring the difference between a first generated image embedding vector of the target image and a second generated image embedding vector of another image in the room with a candidate room shape that matches the shape (e.g., additional images acquired in the room within a defined time period of minutes and / or hours; existing images previously acquired in the room and having a specified location in the room, etc.), identifying better matches with smaller differences, etc.;

[0020] In addition, after determining the initial acquisition location of the target image for each shape match candidate in the manner described above, at least some embodiments include performing one or more additional automatic operations to update and refine the initial acquisition location of one or more shape match candidates (e.g., all shape match candidates; one or more best matches based on a combination of match scores, or if only one match score is used, one or more best matches based on a single match score; etc.), such as one or more of the following non-exclusive lists: using a scored distance between the location of each corner of the estimated room shape of the target image and the candidate room shape of the shape match candidate, using a particular arrangement of the estimated room shape of the target image of the shape match candidate, such as by identifying pairs of corners of two room shapes within a first defined distance threshold (referred to herein as "corner inliers") and refining the acquisition location of the target image using weighted least squares regression (e.g., using confidence scores generated for each corner inliers, such as by a trained neural network, as weights for regression); using a scored distance between the location of each corner of the estimated room shape of the target image and the candidate room shape of the shape match candidate; scoring distances between locations of wireframe structure lines of a shape and locations of wireframe structure lines of a shape-matching alternative candidate room shape, using a particular arrangement of an estimated room shape of a target image using the shape-matching alternative, such as by identifying pairs of horizontal structure lines of two room shapes within a second defined distance threshold (referred to herein as "inline points"), and refining the acquisition locations of the target image using weighted least squares regression (e.g., using confidence scores generated for each inline point, e.g., by a trained neural network, as weights for the regression); performing differentiable rendering optimization using image normal / orthogonal direction predictions by rendering pixel-level surface normal information of the estimated room shape of the target image using an initial acquisition location of the shape-matching alternative (e.g., by using a trained neural network), and comparing those rendered pixel-level surface normal values ​​with other pixel-level surface normal values ​​estimated from existing images having specified locations in the candidate room shape to determine a difference-based cost value, such as by iterating until the cost value reaches a local minimum; and the like.

[0021] In addition, in at least some embodiments, the determined initial and / or updated acquisition locations of the target image can be further refined in conjunction with other determined acquisition locations of other target images acquired in the building, such as to achieve a global optimization of all of these target images (including optionally using additional information about the relative orientations and / or positions between some or all of these target images, such as that which can be reflected by generated inter-target image links, as discussed in more detail elsewhere herein). The following includes additional details about automatic operations that, in at least some embodiments, can be performed by the ILDM system to compare an estimated room shape of a target image of an enclosed room with other candidate room shapes in order to determine one or more matching target rooms of the building (e.g., a single best matching target room), including determining the acquisition location of the target image within each of the matching target rooms.

[0022] Additionally, in some embodiments, the estimated room shape of an image can be matched to other room shapes without using a floor plan. For example, in at least some embodiments, candidate room shapes are generated using estimated room shapes of other images acquired in one or more buildings, such as other additional images captured simultaneously (e.g., within a defined time period, such as one or more minutes or one or more hours) in the same building as the target image. In such embodiments, the location of the target image can be determined relative to the location of the one or more other additional images, including in some embodiments, dynamically doing so while capturing the target image and the additional images, including using the estimated room shapes of the target image and the other additional images to automatically generate a partial or complete floor plan of the building.

[0023] In addition, in some embodiments, for images captured in defined areas other than rooms, the estimated shape of the image of the defined area can be matched to the shapes of other defined areas in various ways. For example, in at least some embodiments, the target image (and optionally additional images) can be acquired outside of one or more buildings, such as in one of multiple separate areas of one or more properties (e.g., house, garden, patio, deck, backyard, side yard, front yard, swimming pool, parking lot, dock, etc.), each of which has a previously or concurrently determined area shape (e.g., 3D shape, 2D shape, etc.), and if so, the acquisition location of the target image (and optionally additional images) can be automatically determined using the shape of such other defined areas as a candidate "room" shape.

[0024] In some embodiments, automatically determining the acquisition position of the target image in the room by the ILDM system may also include additional operations, and the following includes corresponding additional details, including Figure 2A-2W Examples and their associated descriptions.

[0025] The described techniques provide various benefits in various embodiments, including allowing floor plans of multi-room buildings and other structures to be automatically augmented with information about acquisition locations within the building or other structure where images were acquired, including information about distances from the acquisition locations of the images to walls or other objects in the surrounding building or other structure without having or using information from depth sensors or other distance measuring devices. In addition, such automatic techniques allow such image acquisition location information to be determined more quickly than previously available techniques, and with greater accuracy, in at least some embodiments, including by using information acquired from the actual building environment (rather than from plans about how the building should theoretically be constructed), and by being able to capture changes to structural elements that occur after the building is initially constructed. Such described techniques also provide benefits for allowing improved automated navigation of buildings by mobile devices (e.g., semi-autonomous or fully autonomous vehicles) based at least in part on determined image acquisition locations, including significantly reducing the computing power and time used to attempt to otherwise understand the layout of the building. Furthermore, in some implementations, the described techniques can be used to provide an improved GUI in which a user can more accurately and quickly obtain information about a building's interior (e.g., for navigating therein), including in response to a search request, as part of providing personalized information to a user, as part of providing a user with value estimates and / or other information about a building, etc. Various other benefits are also provided by the described techniques, some of which are further described elsewhere herein.

[0026] As described above, in at least some embodiments and scenarios, some or all of the images of the building acquired may be panoramic images, each of which is acquired at one of a plurality of acquisition locations within or around the building, such as generating a panoramic image at each such acquisition location based on one or more videos captured at the acquisition location (e.g., a 360° video shot from a user rotating a smartphone or other mobile device held at the acquisition location), or multiple images captured in multiple directions from the acquisition location (e.g., from a user rotating a smartphone or other mobile device held at the acquisition location; from automatic rotation of the device at the acquisition location, such as on a tripod at the acquisition location; etc.), or simultaneously capturing all image information at a particular acquisition location (e.g., using one or more fisheye lenses), etc. It should be understood that such panoramic images may in some cases be represented in a spherical coordinate system and provide up to 360° coverage around the horizontal and / or vertical axes (e.g., 360° coverage along the horizontal plane and around the vertical axis), while in other embodiments, the acquired panoramic images or other images may include less than 360° vertical coverage (e.g., for images having a ratio of width to height exceeding typical aspect ratios, such as, reaching or exceeding 21:9 or 16:9 or 3:2 or 7:5 or 4:3 or 5:4 or 1:1, including so-called "ultra-wide" lenses and the resulting ultra-wide images). In addition, it should be understood that a user viewing such a panoramic image (or other image with sufficient horizontal and / or vertical coverage so that only a portion of the image is displayed at any given time) can orient the viewing direction within the panoramic image to different orientations to cause different subsets of images (or "views") to be presented within the panoramic image, and in some cases, such a panoramic image can be represented in a spherical coordinate system (including if the panoramic image is represented in a spherical coordinate system and a particular view is being presented, converting the image being presented to a planar coordinate system, such as a stereoscopic image view prior to display). In addition, acquisition metadata regarding the capture of such a panoramic image can be acquired and used in various ways, such as data acquired from an IMU (inertial measurement unit) sensor or other sensor of the mobile device as the mobile device is carried by the user or moved between acquisition locations, non-exclusive examples of such acquisition metadata can include one or more acquisition times; acquisition locations, such as GPS coordinates or other location indications; acquisition directions and / or orientations; relative or absolute acquisition order of multiple images acquired or otherwise associated with a building, etc., and in at least some embodiments and situations, such acquisition metadata can also optionally be used as part of determining the acquisition location of the image, as further discussed below. The following includes additional details regarding automated operation of an apparatus implementing an image capture and analysis (ICA) system involving acquisition of images and optionally acquisition of metadata, including information regarding Figure 1A-1B and Figure 2A-2D .

[0027] As also described above, in various embodiments, the shapes of the rooms of a building may be automatically determined in various ways, including at a time prior to the automatic determination of the acquisition location of a particular image within the building. For example, in at least some embodiments, a mapping information generation manager (MIGM) system may analyze various images acquired in and around a building to automatically determine the room shapes of the rooms of the building (e.g., 3D room shapes, 2D room shapes, etc.) and automatically generate a floor plan of the building. As an example, if multiple images are acquired within a particular room, the images may be analyzed to determine the 3D shape of the room in the building (e.g., reflecting the geometry of the surrounding structural elements of the building), the analysis may include, for example, automatic operations to "register" the camera positions of the images in a common reference frame, thereby "aligning" the images and estimating the 3D positions and shapes of objects in the room, such as by determining features visible in the content of such images (e.g., when the acquisition device captured the particular image, determining the direction and / or orientation of the acquisition device, the path the acquisition device traveled through the room, etc., such as by using SLAM techniques for multiple video frame images and / or other techniques for "dense" sets of images. SfM techniques, the image sets are separated by at most a defined distance (e.g., 6 feet) to generate a 3D point cloud of the room, the room including 3D points along the walls of the room and at least some of the ceiling and floor of the room, and optionally 3D points corresponding to other objects in the room, etc.) and / or by determining and aggregating information about the planes of detected features and normal (orthogonal) directions to those planes to identify planar surfaces of possible locations of walls and other surfaces of the room and connect various possible wall locations (e.g., using one or more constraints, such as, having 90° angles between walls and / or between walls and floor, as part of the so-called "Manhattan world assumption"), and forming an estimated room shape for the room. After determining the estimated room shape of the room in the building, in at least some embodiments, the automatic operation may also include positioning multiple room shapes together to form a floor plan of the building and / or other related mapping information, such as by connecting the various room shapes. The following includes additional details about the automatic operation of an apparatus for implementing a MIGM system, the MIGM system involving determining room shapes and combining room shapes to generate a floor plan, including information about Figure 1A-1B and Figure 2E-2M and elsewhere in this article.

[0028] For the purpose of illustration, some embodiments are described below, in which specific types of information are obtained, used and / or presented in a specific manner for specific types of structures and by using specific types of devices. However, it will be understood that the described technology can be used in other ways in other embodiments, and therefore the invention is not limited to the exemplary details provided. As a non-exclusive example, although a floor plan can be generated for a house that does not include detailed measurements of a specific room or the entire house, it should be understood that in other embodiments, other types of floor plans or other mapping information can be similarly generated, including buildings (or other structures or layouts) separated from the house. As another non-exclusive example, although the floor plan of a house or other building can be used for display to help viewers navigate in the building, the generated mapping information can be used in other ways in other embodiments. In addition, the term "building" refers to any partially or completely enclosed structure in this article, usually but not necessarily including one or more rooms that visually or otherwise separate the interior space of the structure. Non-limiting examples of such buildings include houses, apartment buildings or individual apartments, apartments, office buildings, commercial buildings or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.) therein. As used herein, the terms "acquire" or "capture" with respect to a building interior, acquisition location, or other location (unless the context clearly indicates otherwise) may refer to any recording, storage, or recording of a medium, sensor data, and / or other information related to the space, and / or visual characteristics and / or other perceptible characteristics of the building interior or a subset thereof, for example, by a recording device or by another device that receives information from the recording device. As used herein, the term "panoramic image" may refer to a visual representation based on, including, or separable into a plurality of discrete component images, the plurality of discrete component images originating from substantially similar physical locations in different directions, and the visual representation depicts a larger field of view than any discrete component image depicted individually, the discrete component images including images having sufficiently wide-angle viewing angles from the physical location to include angles exceeding those perceptible from a person's gaze in a single direction (e.g., greater than 120° or 150° or 180°, etc.). As used herein, the term "sequence" of acquisition locations generally refers to two or more acquisition locations, each of which is visited at least once in a corresponding sequence, regardless of whether other non-acquisition locations are visited in between, and regardless of whether the visits to the acquisition locations occur during a single continuous time period, or during multiple different time periods, or by a single user and / or device, or by multiple different users and / or devices. In addition, various details are provided in the drawings and text for exemplary purposes, but are not intended to limit the scope of the invention.For example, the sizes and relative positions of elements in the drawings are not necessarily drawn to scale, with some details omitted and / or greater prominence provided (e.g., by size and positioning) to enhance readability and / or clarity. In addition, the same reference numerals may be used throughout the drawings to identify similar elements or actions.

[0029] Figure 1A is an example block diagram of various computing devices and systems that can participate in the described techniques in some implementations. Figure 1A A panoramic image 165 is shown that has been generated by an interior capture and analysis (“ICA”) system 160, which in this example executes on one or more server computing systems 180, such as with respect to one or more buildings or other structures, and for which inter-image orientation links have been generated for at least some pairs of images. Figure 1B One example of such a linked panoramic image acquisition location 210 for a particular house 198 is shown, as discussed further below, and additional details related to the automatic operation of the ICA system are included elsewhere herein, including with respect to Figure 4 In at least some embodiments, at least some of the ICA system may be executed in part on the mobile image acquisition device 185 (whether in addition to or in lieu of the ICA system 160 on one or more server computing systems 180), as described with respect to Figure 1B The MIGM (Mapping Information Generation Manager) system 160 is further discussed in Figure 1A The system is executed on one or more server computing systems 180 in order to generate and provide building floor plans 155 and / or other mapping related information based on the use of the panoramic images 165 and optionally associated metadata regarding their acquisition and linking. Figure 2M to Figure 2O (referred to herein as "2-0" for clarity) shows an example of such a plan view. As discussed further below, additional details related to the automatic operation of the MIGM system are included elsewhere in this document, including with respect to Figure 5A-Figure 5B .

[0030] Figure 1AAlso shown is an ILDM (Image Location Determination Manager) system 140 that executes on one or more server computing systems 180 to determine the acquisition locations of other target images 145 (e.g., panoramic images) acquired in one or more building rooms, for example with respect to corresponding floor plans 155. In at least some embodiments and scenarios, one or more users of the ILDM client computing devices 105 may further interact with the ILDM system 140 via the network 170, such as to assist in some automated operations of the ILDM system to determine the acquisition locations of the images based at least in part on analysis of the image content, and / or to subsequently use the determined image acquisition location information in one or more other automated manners. Additional details related to the automated operations of the ILDM system are included elsewhere herein, including with respect to Figure 2P-2W and Figure 6A-6B In some embodiments, the ICA system and / or the MIGM system and / or the ILDM system 140 may be executed on the same one or more server computing systems, such as if multiple or all of these systems are operated by a single entity, or otherwise executed in coordination with one another (e.g., some or all of the functionality of these systems are integrated together into a larger system), while in other embodiments, the ILDM system may instead obtain floor plan information and / or other images (e.g., target images, other additional images, etc.) from one or more external sources and optionally store them locally with the ILDM system (not shown) for further analysis and use.

[0031] One or more users (not shown) of one or more client computing devices 175 may also interact with the ILDM system 140 and optionally with the ICA system and / or the MIGM system via one or more computer networks 170, such as to help determine the acquisition location of one or more target images and obtain corresponding determined acquisition location information, and / or obtain and optionally interact with a generated plan view on which one or more additional images have been positioned, and / or obtain and optionally interact with additional information such as one or more associated existing images (e.g., changing between a plan view and a view of a particular image at an acquisition location within or near the plan view; to change the horizontal and / or vertical viewing direction of a corresponding view of a displayed panoramic image, such as to determine the portion of the panoramic image to which the current user viewing direction is directed, etc.). In addition, although in Figure 1AAlthough not shown in the , a floor plan (or a portion thereof) may be linked to or otherwise associated with one or more other types of information, including a floor plan for a multi-story or other multi-level building to have multiple associated sub-floors for interconnecting different floors or levels (e.g., via connecting stairways), linking or otherwise associating a two-dimensional ("2D") floor plan of a building to or otherwise associating a three-dimensional ("3D") rendered floor plan of a building, etc. In addition, although in Figure 1A Not shown, but in some embodiments, the client computing device 175 (or other device, not shown) may receive and use the determined image acquisition location information (optionally in conjunction with the generated floor plan and / or other generated mapping-related information) in other ways, such as, by those devices (e.g., by an autonomous vehicle or other device) to control or assist in automated navigation activities, whether in lieu of or in addition to displaying the generated information.

[0032] exist Figure 1A In the computing environment shown, network 170 can be one or more publicly accessible linked networks, possibly operated by various different entities (e.g., the Internet). In other implementations, network 170 can have other forms, such as a private network (e.g., a corporate network or university network that is completely or partially inaccessible to non-privileged users). In other implementations, network 170 can include private networks and public networks, where one or more private networks can access and / or come from one or more public networks. In addition, network 170 can include various types of wired and / or wireless networks in various situations. In addition, client computing device 175 and server computing system 180 can include various hardware components and stored information, as described below with respect to Figure 3 discussed in more detail.

[0033] exist Figure 1A In an example of , the ICA system 160 may perform automated operations involving generating multiple panoramic images (e.g., each being a 360 degree panorama about a vertical axis) at multiple associated acquisition locations (e.g., in multiple rooms or other locations within a building or other structure, and optionally around some or all of the exterior of the building or other structure), such as for generating and providing a representation of the interior of a building or other structure. The technique may also include analyzing information to determine a relative position / orientation between each of two or more acquisition locations, creating an inter-panorama position / orientation link in the panorama to each of one or more other panoramas based on the determined position / orientation, and then providing information to display or otherwise present the multiple linked panoramic images of the various acquisition locations within the building.

[0034] Figure 1BA block diagram of an exemplary building interior environment is depicted in which a linked panoramic image has been generated and is ready for use in generating and providing a corresponding building floor plan, and for presenting the linked panoramic image to a user. In particular, Figure 1B A building 198 (in this example, house 198) is included having an interior that is captured at least in part via a plurality of panoramic images, such as by a user (not shown) carrying a mobile device 185 with image acquisition capabilities, through the building interior to a series of a plurality of acquisition locations 210. An embodiment of an ICA system (e.g., ICA system 160 on server computing system 180; a copy of some or all of the ICA system executing on a user's mobile device, such as ICA application 155 executing in memory 152 on device 185; etc.) may automatically perform or assist in capturing data representing the building interior, and in some embodiments further analyzing the captured data to generate linked panoramic images that provide a visual representation of the building interior. Although the user's mobile device may include various hardware components, such as one or more cameras or other imaging systems 135, one or more sensors 148 (e.g., gyroscopes 148a, accelerometers 148b, magnetometers or other compasses 148c, etc., such as part of one or more IMUs or inertial measurement units of the mobile device; altimeters; light detectors, etc.), GPS receivers, one or more hardware processors 132, memory 152, displays 142, microphones, one or more external lights, etc., in at least some embodiments, the mobile device may not access or use a device to measure the depth of objects in the building relative to the location of the mobile device, so that the relationship between different panoramic images and their acquisition locations can be determined in part or in whole based on matching elements in different images and / or by using information from other listed hardware components without using any data from any such depth sensor. In addition, although a direction indicator 109 is provided for the viewer's reference, in at least some embodiments, the mobile device and / or the ICA system may not use such absolute direction information, for example, instead determining the relative direction and distance between the panoramic images 210 without considering the actual geographic location or direction.

[0035] In operation, a user associated with a mobile device arrives at a first acquisition location 210A within a first room within a building (in this example, an entryway from exterior door 190-1 to a living room) and, as the mobile device is rotated about a vertical axis at the first acquisition location (e.g., as the user holds the mobile device stationary relative to the user's body, turns his or her body in a circle), captures a view of a portion of the building interior visible from the acquisition location 210A (e.g., some or all of the first room, and optionally small portions of one or more other adjacent or nearby rooms, such as through a doorway, hallway, staircase, or other connecting passage from the first room). Actions of the user and / or mobile device may be controlled or facilitated through the use of one or more programs executing on the mobile device, such as the ICA application system 155 and / or optional viewer 162, control system 147 to manage I / O (input / output) and / or communications and / or networks of device 185 (e.g., receiving instructions from the user and presenting information to the user), etc., and view capture may be performed by recording video at location 210A and / or taking a series of one or more images at location 210A, including capturing visual information that depicts a number of objects or other elements (e.g., structural details) that may be visible in an image (e.g., a video frame) captured from the acquisition location. Figure 1B In the example of , such objects or other elements include various elements that are structurally part of walls (or "wall elements") of rooms of a house, such as doorways 190 and 197 and their doors (e.g., having swing doors and / or sliding doors), windows 196, and inter-wall boundaries (e.g., corners or edges) 195 (including corner 195-1 in the northwest corner of the house 198, corner 195-2 in the northeast corner of the first room, and corner 195-3 in the southwest corner of the first room). In addition, in Figure 1B Such objects or other elements in the example of may also include other elements within the room, such as furniture 191-193 (e.g., recliner 191; chair 192; table 193; etc.), pictures or paintings or televisions or other objects 194 hanging on the wall (such as 194-1 and 194-2), lamps, etc. The user may also optionally provide a text or auditory identifier to be associated with the collection location, such as "entrance" for collection location 210A or "living room" for collection location 210B, although in other embodiments, the ICA system may automatically generate such an identifier (e.g., by automatically analyzing video and / or other recorded information of the building to perform a corresponding automatic determination, such as by using machine learning) or may not use an identifier.

[0036] After the first acquisition location 210A has been adequately captured (e.g., by a full rotation of the mobile device), the user and / or device 185 may proceed to the next acquisition location (such as acquisition location 210B), optionally recording movement data, such as video and / or other data from hardware components (e.g., from one or more IMUs, from cameras, etc.) during movement between the acquisition locations. At the next acquisition location, the mobile device may similarly capture one or more images from that acquisition location. This process may be repeated from some or all rooms of the building, and optionally from the exterior of the building, as shown for acquisition locations 210C-210M. The acquired video and / or other images of each acquisition location are further analyzed to generate a panoramic image of each of the acquisition locations 210A-210M, including, in some embodiments, stitching together multiple component images to create a panoramic image and / or matching objects and other elements in different images. In addition to generating such panoramic images, further analysis may be performed in at least some embodiments to “link” together at least some of the panoramas and their acquisition locations (for illustration purposes, some corresponding lines 215 between example acquisition locations 210A-210C are shown), such as determining relative position information between pairs of acquisition locations that are visible to each other to store corresponding inter-panorama links (e.g., links 215-AB, 215-BC, and 215-AC between acquisition locations 210A and 210B, 210B and 210C, and 210A and 210C, respectively), and in some embodiments and circumstances further linking at least some acquisition locations that are not visible to each other (e.g., link 215-BE between acquisition locations 210B and 210E, not shown).

[0037] Provides information about Figure 1A-1B Various details are provided, but it is to be understood that the details provided are non-exclusive examples included for purposes of illustration and that other embodiments may be performed in other ways without some or all of such details.

[0038] Figure 2A-2W Based on images taken inside a building, such as Figure 1B An example of automatically generating and presenting information about a floor plan of a building by automatically determining the acquisition location of one or more target images captured within the building 198 discussed in the discussion. In particular, Figure 2A An example image 250a is shown, such as in Figure 1B1, a non-panoramic stereoscopic image (or a northeast-facing subset view of a 360 degree panoramic image taken from the acquisition location and formatted in a rectilinear manner) of a living room of a house 198, in which a direction indicator 109a is further displayed to show the northeast direction in which the image was taken. In the illustrated example, the displayed image includes built-in elements (e.g., light fixture 130a), furniture (e.g., chair 192-1), two windows 196-1, and a picture 194-1 hanging on the north wall of the living room. No inter-room passages into or out of the living room (e.g., doorways or other wall openings) are visible in the image. However, multiple room boundaries are visible in image 250a, including a horizontal boundary between a visible portion of the north wall of the living room and the ceiling and floor of the living room, a horizontal boundary between a visible portion of the east wall of the living room and the ceiling and floor of the living room, and an inter-wall vertical boundary 195-2 between the north wall and the east wall.

[0039] Figure 2B continue Figure 2A of examples, and shows that Figure 1B 198 of the present invention, and further displays the direction indicator 109b to show the northwest direction in which the image was taken. In this example image, a small portion of one of the windows 196-1 and a portion of the window 196-2 and the new lighting fixture 130b continue to be visible. In addition, in the image 250b, a portion of the window 196-2 and the new lighting fixture 130b are shown in the image 250b. Figure 2A Horizontal and vertical room boundaries are visible in a .

[0040] Figure 2C continue Figure 2A-2B of examples, and shows that Figure 1B A third stereoscopic image 250c captured in a southwest direction (e.g., from acquisition location 210B) of a living room of house 198 of FIG. 10A further displays direction indicator 109c to show the southwest direction from which the image was captured. In this example image, a portion of window 196-2 continues to be visible, as if in a manner similar to Figure 2A and Figure 2B The example image also shows the living room's inter-room passage, which in this example is the doorway 190-1 ( Figure 1B It should be appreciated that various other stereoscopic images may be acquired from acquisition location 210B and / or other acquisition locations and displayed in a similar manner.

[0041] Figure 2D Shows Figure 1BFurther information 255d of the portion of the house 198, including the living room and other rooms to the east of the living room. Figure 1B and Figure 2A-2C As discussed, in some embodiments, panoramic images may be captured at different locations within a house, such as locations 210A and 210B in the living room, and the corresponding visual content of these images may then be used to determine the room shape of the living room. In addition, in at least some embodiments, additional images may be captured, such as if the device 185 (not shown) is capturing a video or other series of continuous or near-continuous images as it moves through the interior of the house. In this example, the Figure 1B 115, and in particular, a series of locations 215 along the path at which one or more video frame images (or other series of continuous or nearly continuous images) of the surrounding interior of the house may be captured (e.g., if video data is being captured) while the device 185 is being moved. Examples of such locations include capture locations 240a, 240b, and 240c, as well as locations 240c and 240d. Figure 2E-2J . In this example, the positions 215 along the path are shown as being separated by a short distance (e.g., feet, inches, fractions of an inch, etc.), but it will be appreciated that the video capture may be substantially continuous. Therefore, in at least some embodiments, only a subset of such captured video frame images (or other images from a series of continuous or nearly continuous images) may be selected and used for further analysis. Such as, images separated by such distances and / or images separated by a short period of time (e.g., seconds, fractions of a second, multiple seconds, etc.) between their capture, and / or images separated based on other criteria.

[0042] Figure 2E-2J continue Figure 2A-2D , and shows additional information about a living room and about analyzing 360° image frames from video captured along path 155 to determine a possible shape of the room. Although not shown in these figures, similar techniques can be performed on panoramic images captured at two or more of acquisition locations 210A, 210B, and 210C, either in lieu of or in addition to the analysis Figure 2D In particular, Figure 2E Information 255e is included, which shows that the 360° image frame taken from position 240b will share information about various features with the 360° image frame taken from position 240a, although for simplicity, these features are only shown in FIG. Figure 2E A portion of the living room is shown in Figure 2E, an exemplary line of sight 228 is shown from position 240b to various exemplary features in the room, and a similar exemplary line of sight 227 is shown from position 240a to the corresponding features, which illustrates the degree of difference between the views at significantly separated capture locations. Figure 2A Analysis of the image sequence corresponding to position 215 can provide various information about features of the living room, including information about the relevant planes of the features and the normal directions orthogonal to those planes, as further described with reference to Figure 2F-Figure 2I shown.

[0043] In particular, Figure 2F shows information 255f about the northeastern portion of the living room visible in the subset of 360° image frames taken from locations 240a and 240b, and Figure 2G Information 255g is shown regarding the northwest portion of the living room visible in the other subset of 360° image frames taken from locations 240a and 240b, where various features in those portions of the living room are visible in both 360° image frames (e.g., corners 195-1 and 195-2, windows 196-1 and 196-2, etc.). As part of the automated analysis of the 360° image frames using SLAM and / or MVS and / or SfM techniques, information regarding faces 286e and 286f corresponding to portions of the north wall of the living room can be determined based on the detected features, and information 287e and 285f regarding portions of the east and west walls of the living room can similarly be determined based on corresponding features identified in the images. In addition to identifying such face information of detected features (e.g., for each point in the determined sparse 3D point cloud from image analysis), SLAM and / or MVS and / or SfM techniques may also determine possible positions and orientations / directions 220 of one or more images from capture location 240a and possible positions and orientations / directions 222 of one or more images from capture location 240b (e.g., respectively). Figure 2F Positions 220g and 222g of capture positions 240a and 240b, and Figure 2F The optional directions 220e and 222e of the image subsets shown; and Figure 2G The corresponding positions 220g and 222g of the capture positions 240a and 240b, and Figure 2G 220f and 222f). Although Figure 2F and Figure 2GOnly features of a portion of the living room are shown in , but it should be understood that other portions of the 360° image frame corresponding to other portions of the living room can be analyzed in a similar manner to determine possible information about possible planes of various walls of the room and other features in the living room (not shown). In addition, a similar analysis can be performed between some or all other images at locations 215 in the living room that are selected for use, thereby generating various determined feature planes based on the analysis of various images that may correspond to walls of the room.

[0044] Figure 2H continue Figure 2A-2G , and through analysis of 360° image frames captured at locations 240a and 240b, information 255h is shown regarding various determined feature planes that may correspond to the west and north walls of a living room. The illustrated plane information includes a determined plane 286g near or at the north wall (and the corresponding possible location of the north wall), and a determined plane 285g near or at the west wall (and the corresponding possible location of the west wall). As expected, from the different features detected in the analysis of the two 360° image frames, such as, differences in position, angle and / or length, there is a lot of variation in the different determined planes of the north and west walls, resulting in uncertainty as to the actual precise position and angle of each wall. Despite the above, the ...). Figure 2H Not shown in , but as will be appreciated, similar determined feature planes will be similarly detected for other walls of the living room, as well as determined feature planes corresponding to features not along walls (eg, furniture).

[0045] Fig.2I continue Figure 2A-2H , and shows information 255i about additional determined characteristic planes that may correspond to the north and west walls of the living room, from analysis of various additional 360° image frames selected from additional locations 215 in the living room along the path 115. As expected, in this example, analysis of the other images provides even greater variation in the different determined planes of the north and west walls. Fig.2I Additional determined information is also shown, which is used to aggregate information about the various determined characteristic planes to identify possible locations 295a and 295b of the west and north walls, as shown in FIG. Figure 2J The information 255j is shown. In particular, Fig.2IInformation 291a about normal orthogonal directions of some determined feature planes corresponding to the west wall is shown, as well as additional information 290a about those determined feature planes. In an exemplary embodiment, the determined feature planes are clustered to represent hypothesized wall locations for the west wall, and information about the hypothesized wall locations are combined to determine possible wall locations 295a, such as by weighting information from the various clusters and / or underlying determined feature planes. In at least some embodiments, the hypothesized wall locations and / or normal information are analyzed using machine learning techniques to determine final possible wall locations, optionally by further applying assumptions or other constraints (such as 90° corners, e.g., Figure 2H 282, and / or having a flat wall) as part of the machine learning analysis or analysis results. A similar analysis can be performed for the north wall using information 290b about the corresponding determined characteristic planes and additional information 291b about the resulting normal orthogonal directions for at least some of those determined characteristic planes. Figure 2J Possible wall locations 295a and 295b are shown leading to the west and north walls of the living room, respectively.

[0046] Despite Fig.2I Not shown in the figure, but it should be understood that similarly determined characteristic planes and corresponding normal directions of other walls of the living room will be detected and analyzed to determine their possible positions, thereby obtaining an estimated overall room shape of the living room. In addition, similar analysis is performed on each of the rooms of the building to provide an estimated room shape for each of the rooms.

[0047] Figure 2K continue Figure 2A-2J , and shows information 255K about additional information that can be generated from an image in a room and used in one or more ways in at least some embodiments. In particular, images (e.g., video frames) captured in a living room of house 198 can be analyzed to determine a 3D shape of the living room, such as by determining the 3D shape of the living room from a 3D point cloud of features detected in the video frames (e.g., using SLAM and / or SfM and / or MVS techniques). In this example, information 255K is presented in a format similar to Figure 2CAn example portion of such a point cloud for a living room is reflected in the form of image 250c of FIG. 1 , such as corresponding to the northwest portion of the living room in this example (e.g., including the northwest corner 195-1 of the living room, and window 196-1). Such a point cloud can be further analyzed to determine plane regions, such as corresponding to walls, ceilings, floors, etc., and in some cases to detect features, such as windows, doorways, and other room openings. In this example, a first plane region 298 corresponding to the north wall of the living room is identified, and a second plane region 299 corresponding to window 196-1 is further identified. It should be understood that various other walls and other features in the living room and other rooms of the house 198 may be considered similar.

[0048] Figure 2L Additional information 2551 is shown corresponding to positioning the estimated room shapes of the rooms relative to each other based at least in part on connecting inter-room pathways between the rooms and matching room shape information between adjacent rooms after estimated room shapes have been determined for the rooms on the illustrated floor of the house 198. In at least some embodiments, such information can be processed as constraints on the positioning of the rooms, and optimal or otherwise preferred solutions determined for those constraints. Figure 2L Examples of such constraints in include matching 231 channel information connecting adjacent rooms (e.g., in Figure 2E-2J The invention also provides a method of matching the shapes of adjacent rooms 232 to connect the shapes (e.g., as shown in rooms 229d and 229e) with the locations of the corridors detected in the automatic image analysis discussed above, so that the locations of the corridors are co-located, and matching the shapes of adjacent rooms 232 to connect the shapes (e.g., as shown in rooms 229d and 229e). Whether or in lieu of corridor-based constraints and / or room shape-based constraints, various other types of information may be used in other embodiments, such as, for example, precise or approximate dimensions of the overall dimensions of the house (e.g., based on other metadata available about the building, analysis of images from one or more capture locations outside the building, etc.). House exterior information 233 may also be identified and used as a constraint (e.g., based at least in part on automatic identification of corridors and other features corresponding to the exterior of the building (e.g., windows)), such as to prevent another room from being placed at a location that has been identified as being outside the building.

[0049] Figure 2M Continue to Figure 2-O Figure 2A-2L of examples, and shows that Figure 2A-2L The mapping information generated by the analysis types discussed in . In particular, Figure 2MAn exemplary floor plan 230m that may be constructed based on the location of the estimated room shape is shown, in this example, the floor plan 230m includes indications of walls and doorways and windows. In some embodiments, such a floor plan may have further information shown, such as, for example, other features automatically detected by image analysis and / or subsequently added by one or more users. For example, Figure 2N A modified floor plan 230n is shown that includes various types of additional information, such as may be automatically identified through image analysis and added to the floor plan 230m, including one or more of the following types of information: room labels (e.g., "Living Room" for a living room), room dimensions, visual indications of fixtures or appliances or other built-in features, visual indications of the locations of additional types of related and linked information (e.g., existing panoramic and / or stereo images acquired at a specified location that an end user may select for further display; audio annotations and / or recordings that an end user may select for further presentation, etc.), visual indications of doorways and windows, etc. In other embodiments and scenarios, some or all of this type of information may alternatively be provided by one or more MIGM system operator users and / or ICA system operator users. In addition, when the floor plan 230m and / or 230n is displayed to the end user, one or more user-selectable controls may be added to provide interactive functionality as part of a GUI (graphical user interface) screen 255n, thereby indicating the current floor displayed, thereby allowing the end user to select a different floor to be displayed, etc., in which case a corresponding example user-selectable control 228 is added to the GUI. In some embodiments, changes in floors or other levels may also be made directly from the displayed floor plan, such as by selecting a corresponding connecting path (e.g., stairs to different floors), and other visual changes may be made directly from the displayed floor plan by selecting a corresponding displayed user-selectable control (e.g., selecting a control corresponding to a particular image at a particular location and receiving display of that image, whether in lieu of or in addition to a previous display of the floor plan from which the image was selected). In addition, in some embodiments, information for multiple floors (e.g., all floors) will be displayed simultaneously (whether as separate sub-floors for separate floors, or by integrating room connection information for all rooms and floors into a single floor plan that is shown together at one time). It should be understood that in some embodiments various other types of information may be added, in some embodiments some of the types of information shown may not be provided, and in other embodiments visual indications and user selections of linked and associated information may be displayed and selected in other manners.

[0050] Figure 2-O Continue Figure 2A-2N2-O, and shows additional information 265 that can be generated from the automatic analysis techniques disclosed herein, in which example, the additional information 265 is a 2.5D or 3D model floor plan of a house. Such a model 265 can be additional mapping-related information generated based on floor plans 230m and / or 230n, in which additional information about heights is shown in order to show the visual location in the walls of features such as windows and doors. Although not shown in FIG. 2-O, in some embodiments, additional information can be added to the displayed walls, such as from images taken during video capture (e.g., to render and show actual paint, wallpaper, or other surfaces of the house on the rendered model 265), and / or can be used in other ways to add specified colors, textures, or other visual information to walls or other surfaces.

[0051] Figure 2P-Figure 2R continue Figure 2A To the example of Figure 2-O, where Figure 2P Shows Shows Figure 1B Further information 230p of the living room of the house 198 and an indication 289p of the acquisition location of the additional target image, although in this example the acquisition location has not yet been established by automatic operation of the ILDM system. Figure 1B The images are acquired separately ( Figure 1B The image is captured at the acquisition position 210 for generating the floor plan of the building 198, and the target image is acquired at a different position), and in this example is at a time after the floor plan of the building 198 is generated. In this example, the target image is a 360° panoramic image, such as Figure 2Q As shown in the image 250q of FIG. 1 , a spherical format is used to simultaneously display all 360° horizontal of the visual content of the target image (the acquisition position is in the northwest quadrant of the living room, and the starting acquisition direction of the panoramic image is oriented toward the west and is continuous on the 360° horizontal circle). In other embodiments, such a target image may have other forms (e.g., a 180° panoramic image, such as a 360° horizontal circle in a rectilinear format). Figure 2R An example image 255r is shown; a stereo image such as corresponding to Figure 2R d, which in this example is shown as a subset of image 255r; and so on). After providing the target image and optionally acquisition metadata associated with its capture, the ILDM system can perform various automated operations to use the visual content of the target image (and optionally the acquisition metadata) to determine the acquisition location of the target image, including determining that the target image was captured in the living room (rather than in another room of building 198, or in some cases, in other rooms of other buildings) and determining the specific acquisition location and orientation of the target image within the living room, with the acquisition location of the target image so determined subsequently being processed in a manner similar to Figure 2P in a manner overlaid on the floor plan of the building displayed.

[0052] like Figure 2S As shown, automatic operation of the ILDM system can determine 3D room shape information and associated information for a target image by analyzing the visual content of the target image 250q and optionally analyzing its associated acquisition metadata, with example results shown in image 255s and its associated information. In particular, as part of the analysis of the target image, the ILDM system can generate at least a 3D point cloud 271 corresponding to the walls of the room containing the target image, which can be used as some or all of the estimated 3D room shape for the target image. Although in Figure 2S Not shown in the examples of , but in at least some embodiments and cases, such a 3D point cloud may also include some or all of the ceiling and / or floor and / or other structural elements in the room. In addition, the ILDM system may also generate estimated positions of planar surfaces corresponding to at least the walls (and optionally the ceiling and floor), either in addition to or in place of the 3D point cloud, and use the estimated planar surface positions at the estimated 3D room shape of some or all of the target image. Figure 2T Information 238t shows estimated 2D room shape information for walls of an enclosed room of the target image, which can be generated from such a 3D point cloud and / or planar surface. The ILDM system can also generate wireframe structure lines 272 of the enclosed room (e.g., at least the wall boundaries with other walls and ceiling / floor), whether in addition to or instead of the 3D point cloud, and can further use the wireframe structure lines as at least some of the estimated 3D room shape of the target image. In addition, Figure 2S Also shown is that the automatic analysis performed by the ILDM system can identify 3D bounding boxes 273 for identified structural wall elements such as windows and doorways, wherein example bounding boxes are shown for windows on the north wall and a doorway leading from the living room to the hallway, but it should be understood that other windows and doorways (and optionally other structural wall elements) can also have 3D bounding boxes generated similarly. In addition, in at least some embodiments, the ILDM system can also generate labels to correspond to the determined room type of the enclosed room and / or to the determined object type of the structural wall elements identified in the room, such as, the room type label 274 and object type label 276 shown in this example. In addition, the room type label 274 shown in this example has an associated confidence level, and in some cases, multiple room type labels and / or object type labels can be generated for a given room or object with different confidence levels. In addition, in at least some embodiments, the automatic operation of the ILDM system can also include generating an embedding vector that encodes information about elements and / or features of a room in a concise format for later use, such as Figure 2S , (although details of the specific embedding vectors are not shown in this example). In various embodiments, various automatic operations of the ILDM system may be performed in various ways to generate estimated room shape information for a target image, as discussed in more detail elsewhere herein.

[0053] Figure 2T-Figure 2W continue Figure 2A-2S and shows that the ILDM system can automatically determine Figure 2P and Figure 2Q Aspects of further automated operation of the acquisition position of the target image shown. Specifically, Figure 2T Information 255T relating to information about determined room shapes in building 198 is shown for use by the ILDM system as a candidate for estimated room shapes of enclosed rooms that are automatically matched to a target image (and optionally for retrieving determined room shapes of rooms of multiple buildings in other circumstances, such as if the ILDM system further performs a determination of the building in which the target image was acquired). The acquired determined room shapes may be, for example, determined 2D and / or 3D room shapes previously specified for other rooms in the building and / or automatically generated based on 2D room shape outlines on a floor plan of the building, and / or determined 2D and / or 3D room shapes estimated from additional images acquired in building 198, such as during the same time period as acquisition of the target image, and dynamically generated at the same or similar time as the estimated room shape for the target image was generated. In this example, Figure 2T The information 239a, 239b and 239c are shown to correspond to an alternative way of representing the estimated room shape of the target image, where the information 239a corresponds to Figure 2SThe 3D room shape shown in the result image 255s (e.g., showing structural wireframe lines of at least the wall boundaries and showing the locations of wall structural elements such as windows and doorways and other wall openings, but not showing some details that can be determined and used for the 3D room shape, such as the presence of a 3D point cloud, a bounding box, etc.), and information 239b corresponding to the 2D room shape for the enclosed room, which shows the estimated locations of walls and structural elements (such as windows, doorways and other inter-room wall openings), and information 239c corresponding to the 3D room shape for the enclosed room, in a manner similar to information 239b but showing wall heights. In various embodiments, shape matching may be performed using room shape information of type 239a and / or 239b and / or 239c. It should be understood that the estimated room shape information for the enclosed room of the target image may be slightly different from the actual enclosed room, such as having opposing walls of slightly different lengths, having walls that intersect at angles other than 90°, etc. In addition to the estimated room shape information for the enclosed room of the target image, Figure 2T Also shown are determined room shape information for three example rooms of building 198, such as previously determined room shapes associated with a floor plan of the building. In this example, three alternative types of information are shown, a determined room shape for a corridor (i.e., determined room shape information 236a is of a type similar to estimated room shape information 239a, determined room shape information 236b is of a type similar to estimated room shape information 239b, and determined room shape information 236c is of a type similar to estimated room shape information 239c), a determined room shape for a living room (i.e., determined room shape information 237a, 237b, and 237c are of types similar to estimated room shape information 239a, 239b, and 239c, respectively), and a determined room shape for bedroom 1 (i.e., determined room shape information 238a, 238b, and 238c are of types similar to estimated room shape information 239a, 239b, and 239c, respectively). Although in Figure 2T Not shown in FIG. 1 , similar determined room shape information may be retrieved for some or all other rooms of building 198 (and optionally for one or more other buildings, such as other structures on the same property, other buildings in other geographic areas, etc.). Figure 2T Not shown, but in other embodiments, some or all of the candidate room shapes may instead be determined from other additional images (e.g., other panoramic images acquired during the same or similar time frame in the same building), for example by estimating the room shapes of those other additional images and using those estimated room shapes as candidate room shapes for the target image.

[0054] Figure 2UAlso shown is example information 255u regarding how the estimated room shape of the enclosed room of the target image may be compared with the determined room shape information of the candidate objects to identify one or more best matches, which in this example include the best matches based on the room shape information of the candidate objects. Figure 2T The best match 242 of the determined room shape information 237b of the living room is shown. In this example, Figure 2U A small portion of all possible candidate shape matches using the determined room shape information of the rooms of the building 198 as candidate objects is shown. For example, Figure 2U The top row in shows a small number of possible matches of the estimated room shape of the target image with the determined room shape of the hallway (as shown using Figure 2T In particular, each of the top row of examples involves matching the lower right corner of the estimated room shape of the target image with a corner of the determined room shape of the hallway 237, including using vanishing line angle information (e.g., as Figure 2R 257, but in Figure 2UThe invention also provides a method for aligning walls of the estimated and determined room shapes in different arrangements of the determined room shape (not shown in the example) for different alternative shape matches, and for changing the size of at least one of the estimated room shape and the determined room shape of a particular alternative shape match to fit the other. In this example, the first, fifth, and sixth alternative shape matches in the top row (from left to right) involve matching a single corner of the determined room shape of the hallway (its southeast corner) to a single corner of the estimated room shape, but using vanishing line angles to align different possible arrangements of the determined room shape, although other alternative shape matches between the estimated room shape and the determined room shape of the hallway (not shown in the example) will match other corners of the determined room shape of the hallway to the same lower right corner of the estimated room shape using other possible arrangements, and other shape matches between the estimated room shape and the determined room shape of the hallway (not shown in the example) will match other corners of the estimated room shape to some or all of the corners of the determined room shape of the hallway, and in at least one embodiment, some or all of these possible alternative shape matches are still further analyzed using the ILDM system. In a manner similar to the top row, the middle row includes examples of matching the estimated room shape of the target image with the determined room shape information 237b of the living room, where, as with the hallway, only a small portion of the possible alternative shape matches between the estimated room shape and the determined room shape of the living room are shown, each of these examples involving matching the lower right corner of the estimated room shape with a corner of the determined room shape of the living room, with the first and fifth optional shape matches in the middle row (from left to right) involving matching a single corner of the determined room shape of the living room (its southeast corner) with the same corner of the estimated room shape, but with different possible arrangements of the determined room shape using vanishing line angles for alignment, and with changing the size of at least one of the estimated room shape and the determined room shape of a particular alternative shape match to fit the other, and with a variety of other possible alternative shape matches between the estimated room shapes in the determined room shape of the living room, which, although not shown in this example, are nevertheless further analyzed by the ILDM system in at least some embodiments. Figure 2T The bottom row shown in FIG. 1 includes information similar to the top and middle rows, but the estimated room shape matches the determined room shape information 238b for bedroom 1. Although the ILDM system will further analyze some or all of the candidate shape matches to determine one or more best matches for the estimated room shape of the target image, such as with respect to FIG. Figure 2V Further discussion, however Figure 2U The illustrated candidate shape match 242 corresponds to the best match to be determined.

[0055] In particular, Figure 2VAlso shown is information 255v, which shows that when evaluating Figure 2U Some further automatic operations of the ILDM system when matching 242 the candidate shapes of the target image. As discussed in more detail elsewhere herein, various matching criteria can be used to determine the fit between the estimated room shape of the target image and the specific determined room shape of the candidate object having a specified arrangement relative to the estimated room shape, including optionally generating and combining different matching scores for different matching criteria. As an example, the distance between one or more (e.g., all) matching corners of the estimated room shape and the determined room shape can be measured, such as 234b shown for the two corners in the upper left (northwest) corner, where the smaller the distance reflects the better the match (and in some embodiments, the higher the matching score), although other corners are not shown in this example. The distances of their matching corners can be similarly measured and evaluated. As another example, the distance between one or more (e.g., all) corresponding wireframe structure lines of the estimated room shape and the determined room shape can be measured, such as 234c shown for the two wireframes corresponding to the boundary of the north wall to the floor, with the smaller the distance reflecting the better the match (and in some embodiments, the higher the matching score), while other structure lines in this example are not shown, and their distances between corresponding lines can be similarly measured and evaluated. As yet another example, the distance between one or more (e.g., all) corresponding 3D bounding boxes of the estimated room shape and the determined room shape objects, such as 234a shown for the two doorways on the lower left side of the room, can be measured, and a smaller distance reflects a better match (and in some embodiments, a higher match score), although in other embodiments the entire bounding box area can be compared (e.g., using a cross-spanning joint distance measurement), and although other structural element objects are not shown in this example, their distances between corresponding bounding box portions can be similarly measured and evaluated. As another example, the distance between one or more (e.g., all) corresponding walls of the estimated room shape and the determined room shape can be measured, such as 234d shown for the two walls on the top side of the room shape, where a smaller distance reflects a better match (and in some embodiments, a higher match score), although other walls are not shown in this example, their distances between corresponding walls can be similarly measured and evaluated. Various other criteria can be accessed and used, whether in addition to or in place of some or all of the criteria shown, as discussed in more detail elsewhere herein. After evaluating one or more criteria for candidate shape matches for at least some of the candidate room shapes, selecting at least one best matching candidate shape match for further use in determining a capture position of a target image, such as determining a position within the estimated room shape where a position of the target image appears within the determined room shape based on the relative positions of the best matching candidate shape matches arranged so as to correspond to Figure 2VThe location of the acquisition position 289v for the candidate shape matching shown. In addition, once the acquisition position is determined for such a target image, it can be displayed on an updated version of the floor plan of the building, such as, Figure 2W Information on 255w, 230v and 265v is shown in the updated floor plans.

[0056] Already referenced Figure 2A-2W Various details are provided, but it is to be understood that the details provided are non-exclusive examples included for purposes of illustration and that other implementations may be performed in other ways without some or all of such details.

[0057] As a non-exclusive exemplary embodiment, the automatic operation of the ILDM system may include the following actions. Starting from one or more target images having RGB visual data (but without separate depth data), optionally starting from further acquisition metadata with one or more target images, which may include image capture timestamps, camera IMU signals, camera SLAM-based tracking data and / or image room labels (e.g., provided by a user who captured the target image of its enclosed room). In addition, starting from a floor plan in 2D or 3D using a vector format, and optionally having existing images at specified locations and in image projections different from the projection of any target image, and / or starting from additional images in image projections different from the image projection of any target image, the target images have optional further acquisition metadata of the same type.

[0058] Automatic operation can include pre-processing the target image, and optionally additional images, and optionally existing images, to account for camera intrinsic and extrinsic issues where necessary, such as detecting image vanishing lines and vanishing points, extracting (if stereo images) camera focal length and field of view, accounting for the camera's yaw angle relative to vanishing lines present in the image, and re-projecting the image into spherical space (the new camera pose is horizontal relative to the ground plane).

[0059] The automated operation may also include generating geometry predictions for each target image and optionally for each additional image and optionally for each existing image, including the following: estimating a room shape geometry of an indoor structure where the camera is located (e.g., using a convolutional neural network based room shape estimator, such as HorizonNet and DuLaNet, to approximate the room shape geometry as a 3D shape with uniform room height, where the camera can be found at the origin of the shape); optionally predicting image structure lines using an image structure wireframe estimator (e.g., LCNN), and projecting these lines in the image 3D space as room corner candidates; utilizing an on-image object detection algorithm to generate 2D object bounding boxes with labels and object image descriptors, then ray-casting the 2D image bounding boxes onto the previously estimated 3D room shape, and generating footprints of 3D objects to represent their spatial information of the objects, and utilizing a 3D bounding box generation algorithm; optionally generating image embedding vectors (e.g., using a deep neural network model) for later use in comparing image content similarity and image overlap; and optionally labeling the images with one or more room types (e.g., bedroom, kitchen, etc.).

[0060] Automatic operations may also include generating image-image relationships between each target image and one or more additional images and / or existing images, including the following: optionally using a feature-based image matching algorithm, such as SfM, between pairs of images to resolve image angle connections or paired image position information (e.g., which direction in image A connects to which direction in image B); and optionally using a deep learning-based image co-visibility algorithm between pairs of images to determine image content similarity (e.g., for later use, assuming that images sharing high co-visibility scores have a high chance of being spatially close to each other).

[0061] The automated operation may also include retrieving a set of room shape candidates on which to attempt to position each target image in order to determine a precise acquisition location for the target image, the room shape candidates being obtained from existing room shapes associated with the floor plan and / or room shapes estimated for a set of spatially related additional images. Various heuristics may be used to generate binary relationships between a target image pair and additional images or between a target image and regions in an existing floor plan, including the following: using similarities / overlaps between room type labels of the target image and paired images / regions (if available, such as created by an automated image classification algorithm and / or the photographer and / or subsequent annotators) to aggregate a list of preferred candidate room shapes; using temporal relationships between images (if image capture timestamp metadata is available) to retrieve a set of temporally related additional images; using a feature matching-based image alignment algorithm to generate pairwise or grouped image correlations (e.g., image relative angles or binary image correlations); generating pairwise image-to-image correlations using a neural network-based image comparison algorithm; using IMU metadata collected during the image capture process (if available) to give image angular connections; and generating image spatial relationships using a SLAM-based camera tracking algorithm (if SLAM data is available).

[0062] The automatic operation may also include: performing geometric matching on each target image to one or more candidate room shapes to match the estimated room shape of the target image to a corresponding determined room shape of the room on the floor plan or to a corresponding estimated room shape of an additional image, and localizing the target image to a single room shape (e.g., generating one or more camera pose acquisition locations of the target image, optionally together with a confidence score for each camera pose). The automatic operation typically includes the following steps: proposing multiple shape matching options (which are based on the target image camera pose in the candidate room shape space); calculating a score for each (proposed shape matching location) of the proposed camera poses; selecting the camera pose with the highest score or using a threshold to obtain multiple camera poses; and refining the selected one or more camera poses.

[0063] Proposal of various shape matching options may include assuming that if two room shapes are captured by the same camera at the same height (such as for one or more target images and one or more additional images captured simultaneously during the same time period), they have the same scale. The corners of the room shapes are used to generate a set of corner capture options (alternative shape matches) with different shape orientations between the existing room shape of the target image and the candidate room shape. The shape orientation is generated by capturing the horizontal vanishing angle of the target image to the vanishing angles of the paired additional or existing images or candidate room shapes. Therefore, if there are M predicted room corners in the target image, N room corners in the candidate room shape, and 4 vanishing directions from the target image and the paired additional or existing images, then M*N*4 camera poses are proposed for the target image. When 2 images are captured at inconsistent camera heights, the proposed scale and xyz can be generated by selecting 2 control corners from each shape and using the control corners, and the vanishing angle alignment is used to correct the proposed camera angle.

[0064] Calculating a score for each of the proposed camera poses (proposed shape matching locations) may include combining multiple individual scores according to the proposed camera poses (e.g., taking a weighted sum of each individual score, extracting descriptors from each of these items and using a machine learning model to produce a final score, etc.). The separate scores may include one or more of the following: a corner reprojection score, in which the candidate room shape is reprojected into the target image space, the projected room corners from the candidate room shape are compared to the room corners of the existing room shape from the original target image, and each target room corner is matched to its nearest candidate room shape corner, and the distance of each matched corner pair and the number of matches are used to generate a corner reprojection score (e.g., the closer the match, the higher the score); a wireframe structure line reprojection score, in which the structure lines of the candidate room shape are reprojected into the target image space, the projected structure lines from the candidate room shape are compared to the structure lines of the estimated room shape from the target image, and each target image structure line is matched to its nearest candidate room shape structure line, and the distance of each matched structure line pair and the number of matches are used to generate a wireframe structure line reprojection score (e.g., the closer the match, the higher the score); a structural wall element object reprojection score, in which the 3D object bounding box of the candidate room shape from the candidate room shape is reprojected into the target image space in the estimated room space, projected object bounding boxes from the candidate room shapes are compared with object bounding boxes from the estimated room shape from the target image, and each target image object bounding box is matched with its nearest candidate room shape object bounding box; the distance of each matched object bounding box pair is used based on the consistency of intersection spanning union and object type labels; an image angle score, where departure angles / landing angles starting from the target image to the additional / existing image are generated, where separate departure angles / landing angles are also generated for each pair of images using different techniques (e.g., SfM, convolutional neural networks, etc.), and where a score is calculated by comparing these two sets of angles (e.g., the larger the difference, the more penalized the score is); an image content matching score, where image content similarity for a given pair of images is generated (e.g., using a convolutional neural network); and a shape-based boundary intersection score, where the structural walls of the candidate room shapes are re-projected in the 3D space of the target image, and the mismatch between the structural walls of the projected room shape and the structural walls of the target image estimated room shape is used to evaluate the proposed camera pose.

[0065] Refinement of one or more selected camera poses may include using an initial camera pose of a target image from a previous operation (e.g., using corner point matching), and refining the camera pose using a combination of one or more steps. The steps may include one or more of the following steps: performing alignment using corner inliers, wherein a distance threshold is used to filter all matching pairs from a previous corner matching operation within a certain reprojected image distance (where the resulting corner pairs are referred to as corner inliers), and weighted least squares is used to find the best camera position xyz, confidence scores of predicted corners from a target image estimated room shape (e.g., generated by a neural network model) are used as weights in the weighted least squares regression to generate a more accurate camera position than the previous camera pose; performing alignment using line matching of wireframe structure line predictions of a target image and a candidate room shape (e.g., between horizontal lines on a floor), such as using a distance threshold used to filter out all matching lines from a previous line matching operation within a certain reprojected image distance (where the resulting line pairs are referred to as line inliers point), and using a weighted least squares method for finding an optimal camera position xyz, confidence scores of predicted structure lines of an estimated room shape from a target image (e.g., generated by a neural network model) are used as weights in a weighted least squares regression to generate a camera position that is more accurate than a previous camera pose; and performing a differentiable rendering optimization method using image normal prediction, wherein pixel-level surface normal information of a candidate room shape is rendered in a target image space starting from an initial camera pose guess, the rendered surface normals are compared with surface normals estimated from the target image in its image space (e.g., using a neural network-based method such as Taskonomy), and the camera pose is optimized for a lower cost function value; a cost value is calculated, and the camera pose is iteratively optimized until the cost value reaches a local minimum.

[0066] Various details have been provided with respect to this exemplary non-exclusive embodiment, but it is to be understood that the details provided are included for illustrative purposes and other embodiments may be performed in other ways without some or all of these details.

[0067] Figure 3300 and 310. The block diagram of one or more server computing systems 300 that perform the implementation of the ILDM system 340 and one or more server computing systems 380 that perform the implementation of the ICA system 387 and the MIGM system 388. The server computing system and the ILDM system can be implemented using multiple hardware components that form electronic circuits, and the multiple hardware components are suitable and configured to: when in combined operation, perform at least some of the techniques described herein. In the illustrated embodiment, each server computing system 300 includes one or more hardware central processing units ("CPUs") or other hardware processors 305, various input / output ("I / O") components 310, storage 320 and memory 330, and the illustrated I / O components include a display 311, a network connection 312, a computer-readable medium drive 313 and other I / O devices 315 (e.g., keyboards, mice or other pointing devices, microphones, speakers, GPS receivers, etc.). Each server computing system 380 may include hardware components similar to server computing system 300 , including one or more hardware CPU processors 381 , various I / O components 382 , storage 385 , and memory 386 , but some details of server 300 are omitted in server 380 for the sake of brevity.

[0068] The server computing system 300 and the executing ILDM system 340 may communicate via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.) with other computing systems and devices, such as user client computing devices 390 (e.g., for viewing floor plans, associated images, and / or other related information), ICA and MIGM server computing systems 380, one or more mobile image acquisition devices 360, optionally other navigable devices 395 that receive and use floor plans and determined image acquisition locations, and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices), and optionally other computing systems not shown (e.g., for storing and providing additional information related to buildings; for capturing building interior data; for storing and providing information to client computing devices, such as additional supplemental information associated with images and the buildings or other surroundings they contain; etc.).

[0069] In the illustrated embodiment, an embodiment of the ILDM system 340 is executed in the memory 330 to perform at least some of the described techniques, such as by using the processor 305 to execute software instructions of the system 340 in a manner that configures the processor 305 and the computing system 300 to perform automatic operations that implement those described techniques. The illustrated embodiment of the ILDM system may include one or more components not shown to each perform a portion of the functionality of the ILDM system, and the memory may also optionally execute one or more other programs 335. As a specific example, in at least some embodiments, a copy of the ICA and / or MIGM system may be executed as one of the other programs 335, such as executing a copy of the ICA and / or MIGM system instead of or in addition to the ICA system 387 and the MIGM system 388 on the server computing system. The ILDM system 340 may also store and / or retrieve various types of data on the memory 320 (e.g., in one or more databases or other data structures) during its operation, such as various types of floor plan information and other building mapping information 326 (e.g., generated and saved 2D floor plans and the locations of wall elements and other elements on those floor plans, generated and saved 2.5D and / or 3D model floor plans including height information, building and room dimensions for use with associated floor plans, existing images with specified locations, annotation information, etc.), information 321 regarding a target image whose acquisition location is to be determined and associated information 325 regarding such determined acquisition location, information 323 regarding an estimated 3D room shape of the target image and wall element information (e.g., as generated by the ILDM system during its automated operation), user information 328 regarding a user of the client computing device 390 and / or a carrier user of a mobile device 360 ​​interacting with the ILDM, and optionally various other types of additional information 329.The ICA system 387 and / or the MIGM system 388 can similarly store and / or retrieve various types of data on the storage 385 during operation of the storage 385 (e.g., in one or more databases or other data structures), and provide some or all of such information to the ILDM system 340 for its use (whether in a push and / or pull manner), such as images 393 (e.g., acquired 360° panoramic images) and inter-image orientation link information 396 generated by the ICA system and used by the MIGM system to generate a floor plan, final floor plan information generated by the MIGM system and optionally other building mapping information 391 (e.g., similar to or identical to information 326), additional information generated by the MIGM system as part of generating the floor plan, such as determined room shapes 392 and image location information 394, and optionally various types of additional information 397 (e.g., various analytical information related to the display or other use of one or more building interiors or other environments captured by the ICA system).

[0070] Some or all of the user client computing devices 390 (e.g., mobile devices), mobile image acquisition devices 360, other navigable devices 395, and other computing systems may similarly include some or all of the same types of components shown for server computing systems 300 and 380. As a non-limiting example, the mobile image acquisition devices 360 are each shown to include one or more hardware CPUs 361, I / O components 362, storage 365, imaging systems 364, IMU hardware sensors 369, and memory 367, a viewer, and one or both of one or more client applications 368 (e.g., applications specific to the ILDM system and / or ICA system) executed within the memory 367, such as to participate in communications with the ILDM system 340, the ICA system 387, and / or other computing systems. Although specific components are not shown for the other navigable devices 395 or client computing systems 390, it will be appreciated that they may include similar and / or additional components.

[0071] It should also be understood that Figure 3The computing systems 300 and 380 and other systems and devices included in are merely illustrative and are not intended to limit the scope of the invention. The system and / or device may alternatively each include multiple interactive computing systems or devices, and may be connected to other devices not specifically shown, including via Bluetooth communication or other direct communication, via one or more networks, such as the Internet, via a website, or via one or more dedicated networks (e.g., mobile communication networks, etc.). More generally, the device or other computing system may include any combination of hardware that can interact and perform functions of the described type, optionally when programmed or otherwise configured with specific software instructions and / or data structures, including but not limited to desktop computers or other computers (e.g., tablets, tablets, etc.), database servers, network storage devices and other network devices, smart phones and other cellular phones, consumer electronic devices, wearable devices, digital music player devices, handheld gaming devices, PDAs, wireless phones, Internet devices, and various other consumer products including appropriate communication capabilities. Furthermore, in some implementations, the functionality provided by the illustrated ILDM system 340 may be distributed among various components, some of the described functionality of the ILDM system 340 may not be provided, and / or other additional functionality may be provided.

[0072] It should also be understood that, although various items are shown as being stored in memory or stored in storage when in use, for the purpose of memory management and data integrity, these items or their parts can be transmitted between memory and other storage devices. Or in other embodiments, some or all of the software components and / or systems can be executed in a memory on another device and communicate with the computing system shown via inter-computer communication. Therefore, in some embodiments, when configured by one or more software programs (e.g., by the ILDM system 340 executed on the server computing system 300) and / or data structures, for example, by executing the software instructions of one or more software programs and / or by storing such software instructions and / or data structures, and such as executing the algorithms described in the flow charts and other disclosures herein, some or all of the described techniques can be performed by a hardware device including one or more processors and / or memories and / or storage. Furthermore, in some embodiments, some or all of the systems and / or components may be implemented or provided in other ways, such as by being comprised of one or more devices implemented in part or in whole in firmware and / or hardware (e.g., rather than devices implemented in whole or in part by software instructions that configure a particular CPU or other processor), including but not limited to one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), network storage, or portable media articles (e.g., DVD disks, CD disks, optical disks, flash memory devices, etc.) to be read by an appropriate drive or via an appropriate connection. In some embodiments, the systems, components, and data structures may also be transmitted via generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless-based and wired / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Therefore, embodiments of the present disclosure may be implemented with other computer system configurations.

[0073] Figure 4 An example flow chart of an implementation of an ICA system routine 400 is shown. The routine may be implemented by, for example, Figure 1A ICA system 160, Figure 3ICA system 387 and / or ICA system execution as otherwise described herein, such as to acquire 360° panoramic images and / or other images within a building or other structure (e.g., for subsequent generation of associated floor plans and / or other mapping information, such as by implementation of a MIGM system routine, using information about Figure 5A-Figure 5B 4. An example of such a routine is shown. Although portions of the example routine 400 are discussed with respect to acquiring a particular type of image at a particular location, it will be appreciated that this routine or a similar routine may be used to acquire video or other data (e.g., audio) and / or other types of imagery that is not panoramic, either instead of or in addition to such panoramic images. Furthermore, while the illustrated embodiment acquires and uses information from the interior of a target building, it will be appreciated that other embodiments may perform similar techniques on other types of data, including information on non-building structures and / or on the exterior of one or more target buildings of interest. Furthermore, some or all of the routines may be performed on a mobile device used by a user to acquire image information and / or by a system remote from such a mobile device.

[0074] The illustrated implementation of the routine begins at block 405, where instructions or information are received. At block 410, the routine determines whether the received instructions or information indicate acquisition of data representing a building interior, and if not, continues to block 490. Otherwise, the routine proceeds to block 412 to receive an instruction (e.g., from a user of a mobile image acquisition device) to begin an image acquisition process at a first acquisition location. After block 412, the routine proceeds to block 415 to perform acquisition location image acquisition activities to acquire a 360° panoramic image of the acquisition location within the target building interior of interest, such as to provide horizontal coverage of at least 360° around a vertical axis. The routine may also optionally acquire annotations and / or other information from the user regarding the acquisition location and / or surroundings, such as for later use in presenting information regarding the acquisition location and / or surroundings.

[0075] After completion of block 415, the routine continues to block 420 to determine if there are more acquisition locations to acquire images, e.g., based on corresponding information provided by a user of the mobile device. If so, the routine continues to block 422 to optionally initiate the capture of linked information (such as acceleration data, visual data, etc.) during movement of the mobile device along a path of travel away from the current acquisition location and toward a next acquisition location within the building interior. As described elsewhere herein, the captured linked information may include additional sensor data recorded during such movement (e.g., from one or more IMUs or inertial measurement units on the mobile device or other devices carried by the user) and / or additional visual information (e.g., panoramic images, other types of images, panoramic or non-panoramic videos, etc.). Initiating the capture of such linked information may be performed in response to an explicit instruction from a user of the mobile device or based on one or more automatic analyses of information recorded from the mobile device. In addition, in some embodiments, during movement to the next acquisition location, the routine may also optionally monitor the motion of the mobile device and provide one or more guidance prompts to the user regarding the motion of the mobile device, the quality of the sensor data and / or captured visual information, associated lighting / environmental conditions, the desirability of capturing the next acquisition location, and any other suitable aspects of capturing link information. Similarly, the routine may optionally obtain annotations and / or other information from the user regarding the path of travel, such as for later use in presenting information regarding the path of travel or the resulting connection links between panoramic images. At block 424, the routine determines that the mobile device has reached the next acquisition location (e.g., based on an indication from the user, based on the user ceasing forward movement for at least a predetermined amount of time, etc.), used as the new current acquisition location, and returns to block 415 to perform acquisition location image acquisition activities for the new current acquisition location.

[0076] If it is determined in block 420 that there are no more acquisition locations to acquire image information of the current building or other structure, the routine proceeds to block 425 to optionally analyze the acquisition location information of the building or other structure, such as to identify additional coverage (and / or other information) that may be acquired inside the building. For example, the ICA system may provide one or more notifications to the user regarding the information acquired during the capture of multiple acquisition locations and optionally corresponding linked information, such as if it determines that one or more segments of the recorded information are of insufficient or undesirable quality, or do not appear to provide complete coverage of the building. After block 425, the routine continues to block 435 to optionally pre-process the acquired 360° panoramic images before they are subsequently used to generate related mapping information (e.g., placing them in a spherical format, determining vanishing lines and vanishing points of the images, etc.). In block 477, the images and any related generated or acquired information are stored for later use.

[0077] If it is determined in box 410 that the instructions or other information stated in box 405 are not to acquire images and other data representing the interior of a building, the routine instead continues to box 490 to appropriately perform any other indicated operations, such as any housekeeping tasks, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by users of the system, such as users of mobile devices capturing the interior of one or more buildings, operator users of an ICA system, etc.), to acquire and store other information about users of the system, to respond to requests for generated and stored information, and the like.

[0078] After frame 477 or 490, routine proceeds to frame 495 to determine whether to continue, such as until receiving explicit instruction to terminate, or only when receiving explicit instruction to continue. If it is determined to continue, routine returns to frame 405 to wait for additional instructions or information, and if not, then proceeds to step 499 and ends.

[0079] Figure 5A-Figure 5B An exemplary embodiment of a flow chart of a mapping information generation manager (MIGM) system routine 500 is shown. For example, the Figure 1A MIGM system 160, Figure 3 The MIGM system 388 and / or a MIGM system as described elsewhere herein may perform the routine, such as generating a floor plan and optionally other mapping information for the defined area based at least in part on the image of the defined area. Figure 5A-Figure 5BIn the example of , the generated mapping information includes a 2D floor plan and a 3D computer model floor plan of a building (such as a house), but in other embodiments, other types of mapping information can be determined and generated for other types of buildings and used in other ways, as discussed elsewhere herein.

[0080] The illustrated implementation of the routine begins at block 505 where information or instructions are received. The routine continues to block 510 to determine if the instructions received in block 505 indicate that mapping information is to be generated for the indicated building, and if so, the routine continues to execute blocks 515-588 to do so, otherwise continuing to block 590.

[0081] In block 515, the routine determines whether image information is already available for the building, or whether such information needs to be acquired. If it is determined in block 515 that information needs to be acquired, the routine continues to block 520 to acquire such information, optionally waiting for one or more users or devices to move throughout the building and acquire panoramic or other images at multiple acquisition locations in multiple rooms of the building, and optionally further analyzing the acquired images and / or metadata information about them to interconnect the images, as discussed in more detail elsewhere herein. Figure 4 One example implementation of an ICA system routine for performing such image acquisition is provided. If it is determined in block 515 that it is not necessary to acquire images, the routine continues to block 530 to acquire existing panoramic or other images from multiple acquisition locations in multiple rooms of the building, optionally along with interconnection information of the images and metadata information related to movement between acquisition locations, such as may have been provided in block 505 in some cases along with corresponding instructions.

[0082] After box 520 or 530, the routine continues to box 535 to optionally acquire additional information about the building, such as from activities performed during the acquisition and optional analysis of the images and / or from one or more external sources (e.g., an online database, information provided by one or more end users, etc.), such additional information may include, for example, the exterior dimensions and / or shape of the building, additional images and / or annotation information acquired corresponding to specific locations within the building (optionally, for locations different from the acquisition location of the acquired panoramic image or other image), additional images and / or annotation information acquired corresponding to specific locations outside the building, etc. (e.g., surroundings of the building).

[0083] After block 535, the routine continues to block 550, where, for each room in the building having one or more acquisition locations and associated acquired images, a room shape of the room is determined, such as in an automated manner, based on data in the images taken within the room and optionally the specified positions within the room of its acquisition locations. The operations of block 550 may also include using the visual data in the images and / or acquisition metadata for them to determine, for each room in the building, any connecting pathways into and out of the room (e.g., in an automated manner), as well as any wall elements in the room and their locations (e.g., in an automated manner), such as windows, boundaries between walls, etc. The operations of block 550 also include using some or all of the other information determined in block 550 to determine an estimated room shape for the room. After block 550, the routine continues to block 565, where it uses the determined room shapes to create an initial 2D floor plan, such as by connecting the inter-room pathways in their respective rooms, by optionally positioning the room shapes around the determined acquisition locations of the images (e.g., if the acquisition locations are interconnected), and by optionally applying one or more constraints or optimizations. Such a floor plan may include, for example, relative position and shape information for various rooms without providing any actual size information for the individual rooms or the building as a whole, and may also include multiple linked or associated sub-maps of the building (e.g., to reflect different floors, levels, cross-sections, etc.). The routine also associates the locations of doors, wall openings, and other identified wall elements on the floor plan.

[0084] After box 565, the routine optionally performs one or more steps 575-580 to determine additional information and associate the additional information with the floor plan. In box 575, the routine optionally estimates the size of some or all rooms, such as from analysis of the images and / or their acquisition metadata or from overall size information obtained from the exterior of the building, and associates the estimated size with the floor plan. It should be understood that if sufficiently detailed size information is available, architectural drawings, blue prints, etc. can be generated from the floor plan. After box 575, the routine continues to box 580 to optionally associate further information with the floor plan (e.g., with specific rooms or other locations within the building), such as additional existing images with specified locations and / or annotation information. In box 585, the routine further estimates the height of the walls in some or all rooms, such as from analysis of the images and optional sizes of known objects in the images, and height information about the camera when the images were acquired, and further uses this information to generate a 3D computer model floor plan of the building, wherein the 2D and 3D floor plans are associated with each other.

[0085] After box 585, the routine continues to box 588 to store the generated mapping information and optionally other generated information, and optionally further use the generated mapping information, such as to provide a generated 2D floor plan and / or 3D computer model floor plan for display on one or more client devices, provide the generated information to one or more other devices for use in automated navigation of those devices and / or associated vehicles or other entities, etc.

[0086] If it is determined in box 510 that the information or instruction received in box 505 is not to generate mapping information for the indicated building, the routine continues to box 590 to appropriately perform one or more other indicated operations. Such other operations may include, for example, receiving and responding to requests for previously generated floor plans and / or other generated information (e.g., requesting the ILDM system to use such information, requesting such information to be displayed on one or more client devices, requesting such information to be provided to one or more other devices for use in automatic navigation, etc.), obtaining and storing information about the building for use in subsequent operations (e.g., information about size, number or type of rooms, total square footage, other buildings adjacent or nearby, adjacent or nearby vegetation, external images, etc.), etc.

[0087] After frame 588 or 590, routine continues to frame 595, to determine whether to continue, such as until receiving explicit instruction to terminate, or only when receiving explicit instruction to continue. If it is determined to continue, routine returns to frame 505 to wait and receive additional instructions or information, and otherwise continues to frame 599 and ends.

[0088] Figure 6A-6B An exemplary embodiment of a flow chart for an image location determination manager (ILDM) system routine 600 is shown. The routine may be executed by, for example, Figure 1A ILDM system 140, Figure 3 ILDM System 340 and / or reference Figure 2P-2W and the ILDM system described elsewhere herein, such as to perform automatic operations related to the determined acquisition location of the image based at least in part on analysis of the image content and using the associated room shape, and then using the determined acquisition location information in one or more automatic ways. Figure 6A-6B In the example of , the acquisition location is determined relative to a floor plan of a building (such as a house), but in other embodiments, other types of mapping information may be used for other types of structures or for non-structural locations, and the determined acquisition location information may be used in other ways than discussed with respect to routine 600, as discussed elsewhere herein.

[0089] The illustrated implementation of the routine begins at block 605 where information or instructions are received. The routine continues to block 610 to determine if the instructions received in block 605 indicate determining the acquisition location of the target image of the indicated building, and if so, the routine continues to execute blocks 612-688 to do so, otherwise continuing to block 690.

[0090] In block 612, the routine determines whether to use the room shape from a floor plan of the building or from additional images acquired for the building (e.g., concurrently during the same time period as the target image). If from additional images, the routine proceeds to block 617 to retrieve those multiple additional images and add them to the set of images for subsequent 3D room shape estimation determination (unless room shapes have already been estimated for the additional images, in which case those estimated room shapes are retrieved), otherwise proceeds to block 615 to retrieve a floor plan of the building with associated information about the 3D room shapes and room types of the rooms (and optionally including 3D bounding boxes and / or object types for wall structural elements, and / or wireframe structural lines for the room shapes, etc.), and optionally information about existing associated images with specified locations on the floor plan, and select the room shape as a candidate for the acquisition location containing the target image.

[0091] After either block 615 or 617, the routine continues to block 619, where it acquires the target image and optionally acquires metadata for the target image (e.g., one or more of acquisition time, acquisition order with respect to other images acquired for the building, IMU data, SLAM-based tracking data, room type labels, etc.), such as to receive information in block 605 or otherwise retrieve stored information. And the target image is added to the image group for estimated room shape determination. In block 620, the routine then analyzes the visual content of each image in the image group to determine vanishing line and vanishing point information for those images, and if not already performed, does the same for any existing images with the specified location on the floor plan. In box 625, the routine then analyzes the visual content of each image in the image group to estimate the 3D room shape geometry of the room containing the image, including determining a 3D bounding box for structural wall element features, optionally estimating wireframe structural room boundary lines, optionally generating room type labels, optionally generating object type labels for structural wall elements, and optionally generating an image embedding vector to encode information about image features and / or image attributes.

[0092] After block 625, the routine continues to block 640 to determine whether the candidate room shape is from an additional image or from a floor plan, and if the former proceeds to block 645 to select the estimated room shape from the additional image of block 625 to use as a candidate for the acquisition location containing the target image. After block 645, or if it is determined in block 640 that the candidate room shape is from a floor plan, the routine continues to block 650, where, if the image group includes additional images and / or if there are existing images with the specified location on the floor plan, it optionally performs operations including determining the relationship of each pair of target images to one of those other images, such as to determine an angular connection between the images using feature matching of the visual content of the pair of images, and / or to determine overall image similarity of the user's images using common view elements of the images. After block 650, the routine continues to block 655, where it optionally uses acquisition metadata of the target image to identify one or more preferred room shape candidates, such as based on one or more of room type label matching, target image acquisition order, etc. In block 660, the routine then proposes a match of the estimated room shape of the target image to one or more room shape candidates (using preferred candidates if available, such as before or in place of other candidates), including matching corners and using vanishing lines to generate multiple possible candidate shape matches with different candidate room shape arrangements for each room shape candidate. In block 665, the routine then generates a match score for each proposed match, such as by using one or more scoring criteria, and combining their scores if multiple scoring criteria are used, wherein the scoring criteria are optionally based on one or more of corner reprojection, object bounding box reprojection, image angle information, image similarity, and shape-based boundary intersection, wherein one or more best matches are selected based on the scores and used to generate an initial determination of a target image acquisition location (e.g., acquisition location and optionally orientation) within the one or more best matching room shapes.

[0093] In block 670, the routine then generates an updated determination of target image acquisition positions within the one or more best matching room shapes, such as by using one or more of alignment based on points within corners, alignment based on points within wireframe structure lines, and distinguishable presentation using image normal information. In block 685, the routine then optionally optimizes the updated determination of target image acquisition positions within the one or more best matching room shapes in conjunction with other related acquisition position determinations for other images acquired for the building, such as using additional information about the relationship between the images to adjust their final acquisition positions. After block 685, the routine continues to block 688 to store the information determined and generated in blocks 612-685, and optionally display the determined image acquisition position information of the image in its enclosed room on a floor plan (or floor plan excerpt), although in other embodiments, the determined information may be used in other ways (e.g., for automatic navigation of one or more devices).

[0094] If it is determined in block 610 that the information or instructions received in block 605 are not to determine the acquisition location of the target image, the routine instead continues to block 690 to appropriately perform one or more other indicated operations. Such other operations may include, for example, receiving and responding to requests for previously determined image acquisition location information and / or associated target images (e.g., requesting such information to be displayed on one or more client devices, requesting such information to provide such information to one or more other devices for use in automatic navigation, etc.), obtaining and storing information about the building for use in later operations (e.g., information about floor plans and related wall element locations of rooms in the floor plans, etc.), etc.

[0095] After frame 688 or 690, routine continues to frame 695, to determine whether to continue, such as until receiving explicit instruction to terminate, or only when receiving explicit instruction to continue. If it is determined to continue, routine returns to frame 605 to wait and receive additional instructions or information, otherwise continues to frame 699 and ends.

[0096] Figure 7 An exemplary embodiment of a flow chart of a building map viewer routine 700 is shown. The routine may be executed by, for example, Figure 1A The mapping viewer user client computing device 175 and its software system (not shown), Figure 3 The client computing device 390 and / or a mapping information viewer or presentation system as described elsewhere herein is executed, such as to receive and display mapping information (e.g., a 2D or 3D floor plan) for a defined area including visual indications of one or more determined image acquisition locations, and optionally display additional information (e.g., images) associated with specific locations in the mapping information. Figure 7 In the example shown, the mapping information presented is for the interior of a building (such as a house), but in other embodiments, other types of mapping information may be presented for other types of buildings or environments and used in other ways, as discussed elsewhere herein.

[0097] The illustrated implementation of the routine begins at block 705, where an instruction or information is received. At block 710, the routine determines whether the received instruction or information indicates displaying or otherwise presenting information representing the interior of a building, and if not, continues to block 790. Otherwise, the routine proceeds to block 712 to retrieve a floor plan of the building and / or other generated mapping information, and optionally retrieves an indication of associated link information for surrounding locations within the building and / or outside the building, and selects an initial view of the retrieved information (e.g., a view of the floor plan). In block 715, the routine then displays or otherwise presents a current view of the retrieved information, and waits for a user selection in block 717. After the user selection in block 717, if it is determined in block 720 that the user selection corresponds to the current building area (e.g., to change the current view), the routine continues to block 722 to update the current view according to the user selection, and then returns to block 715 to update the displayed or otherwise presented information accordingly. The user selection and corresponding update of the current view may include, for example, displaying or otherwise presenting an associated piece of linked information selected by the user (e.g., a particular image associated with a displayed visual indication of a determined acquisition location), changing the manner in which the current view is displayed (e.g., zooming in or out; rotating the information if appropriate; selecting a new portion of the plan view to be displayed or otherwise presented, such as where some or all of the new portion was not previously visible, or where instead the new portion is a subset of the previously visible information; etc.).

[0098] If it is determined in box 710 that the instruction or other information received in box 705 is not to present information representing the interior of a building, the routine instead continues to box 790 to appropriately perform any other indicated operations, such as any housekeeping tasks, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by users of the system, such as capturing users of mobile devices inside one or more buildings, operator users of an ILDM system, etc.), to obtain and store other information about users of the system, to respond to requests for generated and stored information, and the like.

[0099] After block 790, or if it is determined in block 720 that the user selection does not correspond to the current building area, the routine proceeds to block 795 to determine whether to continue, such as until an explicit indication to terminate is received, or only upon receiving an explicit indication to continue. If it is determined to continue (e.g., if the user makes a selection related to a new location to be presented in block 717), the routine returns to block 705 to wait for additional instructions or information (or if the user makes a selection related to a new location to be presented in block 717, then proceed to block 712), and if not, proceed to step 799 and end.

[0100] Non-exclusive exemplary embodiments described herein are further described in the following clauses.

[0101] A01. A computer-implemented method comprising:

[0102] receiving, by one or more computing systems, a target image acquired in a building having a plurality of rooms, wherein the target image is a panoramic image having a visual overlay of a plurality of surrounding walls;

[0103] Determining, by the one or more computing systems, a capture location of the target image on the floor plan of the building, comprising:

[0104] obtaining, by the one or more computing systems, determined three-dimensional ("3D") room shapes of the plurality of rooms;

[0105] generating, by the one or more computing systems and based at least in part on analyzing visual content of the target image, an estimated 3D room shape visible in the target image and an indicated location within the estimated 3D room shape of the target image;

[0106] selecting, by the one or more computing systems, one of a plurality of candidate shape matches for the estimated 3D room shape based at least in part on the degree of match determined for the selected one candidate shape match, wherein the plurality of candidate shape matches each include an arrangement of one of the determined 3D room shapes relative to the estimated 3D room shape and have a determined degree of match based at least in part on a fit of the one determined 3D room shape to the arrangement of the estimated 3D room shape, and wherein the one determined 3D room shape of the selected one candidate shape match is an identified room of the plurality of rooms;

[0107] locating, by the one or more computing systems, the indicated position with respect to the arrangement within the estimated 3D room shape using the determined 3D room shape relative to the selected one of the candidate shapes matching the estimated 3D room shape, and identifying, on the floor plan, the acquisition position of the target image based on the located indicated position within the identified room; and

[0108] Information regarding the determined acquisition location of the target image in the identified room is provided by the one or more computing systems to enable display of the determined acquisition location on the floor plan of the building.

[0109] A02. A computer-implemented method according to clause A01, wherein determining the acquisition location of the target image on the floor plan includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location, the orientation identifying a direction in at least two dimensions of a starting view starting from at least the determined acquisition location, and wherein providing the information includes presenting, by the one or more computing systems and on a client device, a display of at least some of the floor plan of the building, the display including a visual indication of the identified room and overlaying the determined acquisition location and a visual indication of the determined orientation.

[0110] A03. A computer-implemented method as described in any of clauses A01-A02, wherein generating the estimated 3D room shape comprises:

[0111] determining, by the one or more computing systems, room shape information of surrounding rooms surrounding the acquisition location of the target image, the room shape information comprising at least one of a 3D point cloud modeling at least some walls of the surrounding room, or a planar surface representing at least some walls of the surrounding room, or a wireframe structural line representing a boundary of at least some walls of the surrounding room, and further comprising structural wall elements of the surrounding room, the structural wall elements comprising at least windows, doorways, and non-doorway wall openings, and

[0112] The determined room shape information using the estimated 3D room shape.

[0113] A04. A computer-implemented method as described in clause A03, wherein generating the estimated 3D room shape includes using information about the acquisition time of the target image and also includes determining, by the one or more computing systems, additional room shape information, the additional room shape information comprising at least one of: a 3D bounding box surrounding the structural wall elements, or at least one room type label for the surrounding rooms, or at least one object type label for each of the structural wall elements, or at least one embedding vector generated by a trained neural network for the target image for encoding information about features of the surrounding rooms.

[0114] A05. A computer-implemented method as described in any of clauses A03-A04, wherein determining the room shape information of the surrounding room includes analyzing the visual content of the target image by the one or more computing systems and using one or more trained neural networks.

[0115] A06. A computer-implemented method as described in any of clauses A01-A05, wherein determining the acquisition position of the target image also includes determining, by the one or more computing systems, the plurality of candidate shape matches for the estimated 3D room shape, comprising:

[0116] selecting, by the one or more computing systems, one or more of the determined 3D room shapes as a candidate room shape for matching with the estimated 3D room shape; and

[0117] generating, by the one or more computing systems and for each of the selected candidate room shapes, for each of the one or more candidate shape matches, one or more candidate shape matches of the plurality of candidate shape matches by matching one or more corners of the selected candidate shape with one or more corners of the estimated 3D room shape, and aligning one or more walls of the selected candidate room shape with one or more walls of the estimated 3D room shape by using vanishing line angle information of the target image, and determining an arrangement of the selected candidate room shape relative to the estimated 3D room shape by adjusting sizes of at least one of the estimated 3D room shape or the selected candidate room shapes to reflect each other.

[0118] A07. The computer-implemented method of clause A06, wherein the selecting the one or more determined 3D room shapes comprises using information about an acquisition time of the target image, and further comprising identifying, by the one or more computing systems, a preferred subset of the determined 3D room shapes to use as candidate room shapes for the selection, comprising using at least one of:

[0119] a room type label of a match between the estimated 3D room shape and the selected candidate room shape; or

[0120] one or more determined 3D room shapes of one or more rooms of the building associated with one or more additional images acquired before and / or after the target image in the building; or

[0121] one or more determined 3D room shapes of one or more rooms of the building associated with one or more additional images having features matching features of the target image above a first defined matching threshold; or

[0122] one or more determined 3D room shapes of one or more rooms of the building associated with one or more additional images having an overall similarity to the target image by matching a common visible area above a second defined matching threshold; or

[0123] One or more determined 3D room shapes of one or more rooms of the building associated with one or more additional images, wherein a plurality of determined directions between each pair of the target images and one of the additional images matches above a third defined match threshold, wherein one of the plurality of determined directions between each pair is based on visual content of the pair of images, and another one of the plurality of determined directions between each pair is based at least in part on acquisition metadata from acquisition of the target images.

[0124] A08. A computer-implemented method as described in any of clauses A06-A07, wherein the selecting the one or more determined 3D room shapes includes selecting each of the determined 3D room shapes as a candidate room shape, and wherein generating the one or more alternative shape matches for each of the selected candidate room shapes includes generating a separate alternative shape match for each combination of a corner of the selected candidate room shape with a corner of the estimated 3D room shape and a vanishing line angle of the target image.

[0125] A09. A computer-implemented method as described in any of clauses A06-A08, wherein the selecting one of the plurality of alternative shape matches for the estimated 3D room shape based at least in part on the determined degree of match for the selected one alternative shape match comprises: determining, by the one or more computing systems and for the selected one alternative shape match, a match score between the estimated 3D room shape of the selected one alternative shape match and the determined 3D room shape in the arrangement of the selected one alternative shape match based on one or more matching criteria, wherein the one or more matching criteria include at least one of the following:

[0126] a determined distance between matching corners of the estimated 3D room shape and the determined 3D room shape in the arrangement; or

[0127] a determined distance between matching structural wall locations of said estimated 3D room shape and said determined 3D room shape in said arrangement; or

[0128] a determined distance between matching wireframe structure lines of the estimated 3D room shape and the determined 3D room shape in the arrangement; or

[0129] a determined distance between said estimated 3D room shape and matching 3D bounding boxes of structural wall elements in said determined 3D room shape in said arrangement; or

[0130] A determined distance between the estimated 3D room shape and a matched 3D bounding box of a floor element in the determined 3D room shape in the arrangement.

[0131] A10. A computer-implemented method as described in clause A09, wherein the determination of the match score is further performed for each of the multiple alternative shape matches and includes combining separate scores for multiple matching criteria, and wherein the selection of one of the alternative shape matches for the estimated 3D room shape is also based at least in part on the selected one of the alternative shape matches having the highest determined match score.

[0132] A11. The computer-implemented method of any of clauses A09-A10, further comprising updating, by the one or more computing systems, an arrangement of the one determined 3D room shape relative to the selected one candidate shape match of the estimated 3D room shape based on at least one of:

[0133] using least squares regression to reduce a determined distance between said estimated 3D room shape of said selected one candidate shape match and said matching corners of said determined 3D room shape; or

[0134] using least squares regression to reduce a determined distance between said estimated 3D room shape of said selected one candidate shape match and said matched structural wall locations of said determined 3D room shape; or

[0135] using least squares regression to reduce a determined distance between said estimated 3D room shape of said selected one candidate shape match and said matched wireframe structure lines of said determined 3D room shape; or

[0136] using least squares regression to reduce a determined distance between said estimated 3D room shape of said selected one candidate shape match and said matched 3D bounding boxes of structural wall elements in said determined 3D room shape; or

[0137] using least squares regression to reduce a determined distance between said estimated 3D room shape of said selected one candidate shape match and said matched 3D bounding boxes of floor elements in said determined 3D room shape; or

[0138] performing a differentiable rendering optimization on a difference between predicted pixel-level surface normal information for the estimated 3D room shape and other pixel-level surface normal information estimated from existing images having a specified position in the identified room; or

[0139] A differentiable rendering optimization is performed on the difference between predicted pixel-level surface fragment information of the estimated 3D room shape and other pixel-level surface fragment values ​​estimated from existing images having specified positions in the identified room, wherein the pixel-level surface fragment information of the estimated 3D room shape is predicted based on at least one of fragment surface normal information, or fragment surface depth map information, or an output of a convolutional neural network.

[0140] A12. A computer-implemented method as described in any of clauses A01-A11, wherein determining the acquisition location of the target image further comprises: generating, by the one or more computing systems, the plurality of candidate shape matches for the estimated 3D room shape, for each of one or more of the determined 3D room shapes, generating one or more of the plurality of candidate shape matches for the determined 3D room shape, and for each of the one or more candidate shape matches, performing at least one of the following:

[0141] determining a placement of the determined 3D room shape relative to the estimated 3D room shape based at least in part on matching one or more first bounding boxes of one or more first objects on walls of the determined 3D room shape with one or more second bounding boxes of one or more second objects on walls of the estimated 3D room shape; or

[0142] determining a placement of the determined 3D room shape relative to the estimated 3D room shape based at least in part on matching one or more third bounding boxes of one or more third objects on the floor of the determined 3D room shape with one or more fourth bounding boxes of one or more fourth objects on the floor of the estimated 3D room shape; or

[0143] determining an arrangement of the determined 3D room shape relative to the estimated 3D room shape based at least in part on matching one or more first planar surfaces of the determined 3D room shape with one or more second planar surfaces of the estimated 3D room shape; or

[0144] An arrangement of the determined 3D room shape relative to the estimated 3D room shape is determined based at least in part on matching a first 3D point of the determined 3D room shape with a second 3D point of the estimated 3D room shape.

[0145] A13. A computer-implemented method as described in any of clauses A01-A12, wherein selecting one of the candidate shape matches of the estimated 3D room shape based at least in part on the determined degree of match of the estimated 3D room shape comprises determining, by the one or more computing systems and for the selected one candidate shape match, a match score based on one or more matching criteria, wherein the one or more matching criteria comprises at least one of:

[0146] a match above a first defined matching threshold of a plurality of determined directions between the target image and an additional image associated with the selected one candidate shape determined 3D room shape, wherein one of the plurality of determined directions is based on visual content of the target image and the additional image, and wherein another of the plurality of determined directions is based at least in part on acquisition metadata from acquiring the target image; or

[0147] a match above a second defined match threshold between a first estimated orientation of a target image based on visual content of the target image and a second estimated orientation of the target image based at least in part on acquisition metadata from acquisition of the target image; or

[0148] a match above a third defined match threshold between a first estimated position of the target image obtained based on visual content of the target image and a second estimated position of the target image obtained based at least in part on acquisition metadata from an acquisition of the target image; or

[0149] A match above a fourth defined match threshold between a third estimated position of the target image acquired based on visual content of the target image and a fourth estimated position of the target image to be acquired.

[0150] A14. A computer-implemented method as described in any of clauses A01-A13, wherein the target image is one of a plurality of target images having different acquisition positions in the building and each comprising 360 degrees of horizontal visual coverage about a vertical axis, and wherein the method further comprises:

[0151] determining, by the one or more computing systems, a capture location on the floor plan of each of the plurality of target images, including performing the generation of the estimated 3D room shape for each of the plurality of target images without using any depth information from any depth sensing sensor of distances from the capture location of the target image to surrounding surfaces; and

[0152] The determined acquisition positions of at least some of the plurality of target images are updated by performing a global optimization on the determined acquisition positions of the plurality of target images according to the additional information about the relationship between the plurality of target images.

[0153] A15. A computer-implemented method comprising the automatic operation of one or more computing devices, comprising at least:

[0154] For a target image acquired in a building having multiple rooms, determining a collection position of the target image in the building includes:

[0155] for each of one or more of the plurality of rooms, obtaining a determined room shape of the room;

[0156] generating an estimated room shape for the target image based at least in part on analyzing visual content of the target image;

[0157] selecting one of a plurality of candidate shape matches of the estimated room shape based at least in part on a degree of match determined for the selected one candidate shape match, wherein the plurality of candidate shape matches each include an arrangement of one of the acquired determined room shapes of one of the plurality of rooms relative to the estimated room shape and have a determined degree of match based at least in part on a fit of the estimated room shape to the arrangement of that acquired determined room shape;

[0158] Based on the arrangement of the selected one candidate shape matching, and using the information of the target image in the estimated room shape, locating a capture position of the target image in the one room in a room whose determined room shape is obtained for the selected one candidate shape matching; and

[0159] Information about the determined acquisition position of the target image in the one room of the building is provided, and the obtained determined room shape is used for the selected one candidate shape matching.

[0160] A16. A computer-implemented method according to any of clauses A01-A15, wherein determining the acquisition location of the target image in the building includes determining the orientation of the target image at the determined acquisition location, the orientation identifying the direction in at least two dimensions of a starting view from at least the determined acquisition location, and wherein providing the information includes presenting a display of at least some of the building on a client device, the building including the identified room and overlaid with a visual indication of the determined acquisition location and the determined orientation.

[0161] A17. The computer-implemented method of any of clauses A01-A16, wherein, for each of the one or more rooms, obtaining the determined room shape for each of the one or more rooms comprises:

[0162] acquiring at least one additional panoramic image captured in the room; and

[0163] The determined room shape is generated for the room based at least in part on analyzing visual content of the at least one additional panoramic image acquired in the room.

[0164] A18. A computer-implemented method comprising the automatic operation of one or more computing devices, the automatic operation comprising at least:

[0165] receiving, by the one or more computing systems, a target image and a plurality of additional images acquired in a building having a plurality of rooms, wherein each of the target image and the plurality of additional images is a panoramic image having a visual overlay of a plurality of surrounding walls;

[0166] Determining, by the one or more computing systems, a capture location of the target image in the building, comprising:

[0167] acquiring, by the one or more computing systems, three-dimensional (“3D”) room shapes determined from the plurality of additional images, wherein, for an associated one of the plurality of additional images, each determined 3D room shape identifies at least walls and corners visible in a surrounding room;

[0168] generating, by the one or more computing systems and based at least in part on analyzing visual content of the target image, an estimated 3D room shape and an indicated location of the target image within the estimated 3D room shape, the estimated 3D room shape including at least walls and corners visible in the target image;

[0169] selecting, by the one or more computing systems, one of a plurality of candidate shape matches for the estimated 3D room shape based at least in part on the degree of match determined for the selected one candidate shape match, wherein the plurality of candidate shape matches each include an arrangement of one of the determined 3D room shapes relative to the estimated 3D room shape and have a determined degree of match based at least in part on a fit of the walls and corners of the estimated 3D room shape using the candidate shape match with the arrangement of the walls and corners of the one determined 3D room shape of the candidate shape match;

[0170] and, by the one or more computing systems and within the surrounding room, for the additional images associated with the one determined 3D room shape that matches the selected one candidate shape, locating the indicated position within the estimated 3D room shape using the arrangement of the selected one candidate shape matches, and identifying the acquisition position of the target image within the building based on the located indicated position within the surrounding room;

[0171] Information about a capture location of the determined target image within the surrounding room is provided by the one or more computing systems to enable the determined capture location to be displayed using information about the building.

[0172] A19. A computer-implemented method according to any of clauses A01-A18, wherein determining the acquisition location of the target image in the building includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location, the orientation identifying a direction in at least two dimensions of a starting view starting from at least the determined acquisition location, and wherein providing the information includes presenting, by the one or more computing systems and on a client device, a display of at least some of the building, the display including the surrounding rooms and overlaid with a visual indication of the determined acquisition location and a visual indication of the determined orientation.

[0173] A20. A computer-implemented method according to any of clauses A01-A19, wherein said obtaining said determined 3D room shape comprises, for each of one or more rooms in said building, generating said determined 3D room shape of said room by analyzing visual content of at least one of said plurality of additional images acquired in said room, wherein determination of a capture location of said target image is performed dynamically when acquiring said target image, and wherein said providing said information comprises presenting a display of at least some of said building including said surrounding rooms by said one or more computing systems and when acquiring said target image.

[0174] A21. A computer-implemented method as described in any of clauses A01-A20, wherein selecting one of the alternative shape matches for the estimated 3D room shape based at least in part on the determined degree of match of the selected one alternative shape match comprises: determining, by the one or more computing systems and for the selected one alternative shape match, a match score between the estimated 3D room shape of the selected one alternative shape match and the determined 3D room shape in the arrangement of the selected one alternative shape match based on one or more matching criteria, wherein the one or more matching criteria include at least one of the following:

[0175] the distance between matching corners of the estimated 3D room shape and the determined 3D room shape in the arrangement; or

[0176] a distance between matching structural walls of the estimated 3D room shape and the determined 3D room shape in the arrangement; or

[0177] a distance between matching wireframe structure lines of the estimated 3D room shape and the determined 3D room shape in the arrangement; or

[0178] a distance between matching 3D bounding boxes of structural wall elements in said estimated 3D room shape and said determined 3D room shape in said arrangement; or

[0179] a distance between matching 3D bounding boxes of floor elements in the estimated 3D room shape and the determined 3D room shape in the arrangement; or

[0180] for said selected one candidate shape matching said determined 3D room shape, a difference between the visual content of said target image and the visual content of an additional image associated with said surrounding room; or

[0181] A difference between two measurements of direction between the target image and the one additional image, the two measurements comprising a first measurement based on the visual content of the target image and the visual content of the one additional image and a second measurement based at least in part on acquisition metadata from acquiring the target image and the one additional image.

[0182] A22. A computer-implemented method as described in any of clauses A01-A21, wherein selecting one of the candidate shape matches of the estimated 3D room shape based at least in part on the determined degree of match of the estimated 3D room shape comprises determining, by the one or more computing systems and for the selected one candidate shape match based on one or more matching criteria, a matching score, wherein the one or more matching criteria comprises at least one of:

[0183] a match above a first defined match threshold between a first estimated orientation of a target image based on visual content of the target image and a second estimated orientation of the target image based at least in part on acquisition metadata from acquisition of the target image; or

[0184] a match above a second defined match threshold between a first estimated position of the target image obtained based on visual content of the target image and a second estimated position of the target image obtained based at least in part on acquisition metadata from an acquisition of the target image; or

[0185] A match above a third defined match threshold between a third estimated position of the target image acquired based on visual content of the target image and a fourth estimated position of the target image to be acquired.

[0186] A23. A computer-implemented method according to any of clauses A01-A22, wherein selecting one of the multiple alternative shape matches for the estimated 3D room shape based at least in part on a determined degree of match for the selected one alternative shape match includes determining the degree of match by the one or more computing systems and for the selected one alternative shape match based at least in part on comparing a first generated image embedding vector of the target image with a second generated image embedding vector of the additional image associated with the surrounding room of the determined 3D room shape that matches the selected one alternative shape, wherein the first generated image embedding vector encodes information about one or more features and / or attributes of the target image, and wherein the second generated image embedding vector encodes information about one or more features and / or attributes of the additional image.

[0187] A24. A computer-implemented method comprising:

[0188] Receiving, by one or more computing systems, a target image acquired in a building having a plurality of rooms, wherein the target image is a panoramic image including 360 degrees of horizontal visual coverage about a vertical axis;

[0189] Determining, by the one or more computing systems, a capture location of the target image on the floor plan of the building, comprising:

[0190] determining, by the one or more computing systems, three-dimensional ("3D") room shapes for the plurality of rooms, wherein each determined 3D room shape identifies at least walls and corners of an associated one of the plurality of rooms;

[0191] generating, by the one or more computing systems and based at least in part on analyzing visual content of the target image, an estimated 3D room shape for the target image, the estimated 3D room shape identifying at least walls and corners visible in the target image, and an indicated location within the estimated 3D room shape for the target image, and information regarding a vanishing line angle for the target image, the vanishing line angle being based at least in part on walls visible in the target image;

[0192] determining, by the one or more computing systems, a plurality of candidate shape matches for the estimated 3D room shape, the plurality of candidate shape matches each comprising a placement of one of the determined 3D room shapes relative to the estimated 3D room shape, the placement using vanishing line angles of the target image and additional vanishing line angles associated with the one determined 3D room shape to match at least one wall and corner of the estimated 3D room shape with at least one other wall and corner of the one determined 3D room shape;

[0193] selecting, by the one or more computing systems, one of a plurality of candidate shape matches based at least in part on a measure of fit of the selected one candidate shape match of the walls and corners of the estimated 3D room shape with the walls and corners of a determined 3D room shape of an arrangement thereof using the selected one candidate shape match, and identifying a room of the plurality of rooms whose determined room shape is a determined 3D room shape of the selected one candidate shape match;

[0194] locating, by the one or more computing systems and within the identified room, the indicated position within the estimated 3D room shape on the arrangement using the one determined 3D room shape relative to the selected one candidate shape matching arrangement of the estimated 3D room shape, and identifying the acquisition position of the target image on the floor plan based on the located indicated position within the identified room; and

[0195] A floor plan of the building is displayed, by the one or more computing systems, with a visual indication of a capture location of the determined target image in the identified room.

[0196] A25. A computer-implemented method according to any one of clauses A01-A24, wherein determining the 3D room shapes of the multiple rooms includes obtaining two-dimensional ("2D") outlines of the multiple rooms from a floor plan of the building, and generating the 3D room shapes of the multiple rooms based at least in part on the obtained 2D outlines and information about wall heights of the multiple rooms.

[0197] A26. A computer-implemented method according to any one of clauses A01-A25, wherein determining the acquisition position of the target image on the plan view includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition position, the orientation identifying a direction in at least two dimensions of a starting view starting from at least the determined acquisition position, and wherein displaying the plan view also includes displaying a visual indication of the determined orientation by the one or more computing systems and on a client device at the determined acquisition position.

[0198] A27. A computer-implemented method comprising the steps of performing a plurality of automated operations that implement techniques substantially as disclosed herein.

[0199] B01. A non-transitory computer-readable medium having stored executable software instructions and / or other storage contents, wherein the stored executable software instructions and / or other storage contents enable one or more computing systems to perform automatic operations to implement the method of any of clauses A01-A27.

[0200] B02. A non-transitory computer-readable medium having stored executable software instructions and / or other stored content, which enables one or more computing systems to perform automatic operations that implement the technology substantially as disclosed herein.

[0201] C01. One or more computing systems comprising one or more hardware processors and one or more memories having stored instructions, which when executed by at least one of the one or more hardware processors cause the one or more computing systems to perform automatic operations implementing a method of any of clauses A01-A27.

[0202] C02. One or more computing systems comprising one or more hardware processors and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automatic operations that implement the techniques substantially as disclosed herein.

[0203] D01. A computer program adapted to perform the method of any of clauses A01-A27 when said computer program is run on a computer.

[0204] Aspects of the present disclosure are described herein with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure.It should be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer-readable program instructions.It will be further understood that in some implementations, the functions provided by the routines discussed above can be provided in an alternative manner, such as splitting or merging into fewer routines between more routines.Similarly, in some implementations, the routines described can provide more or less functions than described, such as when other routines described lack or include such functions respectively, or when the amount of functions provided changes.In addition, although various operations can be shown as being performed in a particular manner (e.g., serial or parallel, or synchronous or asynchronous) and / or in a particular order, in other implementations, operations can be performed in other orders and other ways.Any data structure discussed above can also be constructed in different ways, such as by dividing a single data structure into multiple data structures and / or by merging multiple data structures into a single data structure. Similarly, in some implementations, illustrated data structures may store more or less information than depicted, such as when other illustrated data structures lack or include such information, respectively, or when the amount or type of information stored varies.

[0205] It will be appreciated from the foregoing that, although specific embodiments are described herein for purposes of illustration, various modifications may be made without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited except for the corresponding claims and the elements recorded in those claims. In addition, although certain aspects of the present invention may be presented in the form of certain claims at certain times, the inventors contemplate various aspects of the present invention in the form of any available claims. For example, although only some aspects of the present invention may be described as embodied in a computer-readable medium at a particular time, other aspects may also be so embodied.

Claims

1. A system comprising: one or more hardware processors of one or more computing devices; as well as one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations, the automated operations comprising at least: For a target image acquired in a building having multiple rooms, determining a collection position of the target image in the building includes: For one or more rooms in the plurality of rooms, obtaining a determined three-dimensional room shape; generating an estimated three-dimensional room shape for the target image based at least in part on analyzing visual content of the target image; For the estimated 3D room shape, multiple candidate shape matches are determined, including: selecting one or more of the determined three-dimensional room shapes as a candidate room shape for matching with the estimated three-dimensional room shape; and for each candidate room shape, for each of one or more candidate shape matches in the plurality of candidate shape matches, generating one or more candidate shape matches in the plurality of candidate shape matches by matching one or more corners of the candidate room shape with one or more corners of the estimated three-dimensional room shape, and aligning one or more walls of the candidate room shape with one or more walls of the estimated three-dimensional room shape by using vanishing line angle information of the target image, and determining an arrangement of the candidate room shape relative to the estimated three-dimensional room shape by adjusting the size of at least one of the estimated three-dimensional room shape or the candidate room shape to reflect each other; selecting one of the plurality of candidate shape matches for the estimated three-dimensional room shape based at least in part on the determined degree of match of the selected one candidate shape match, wherein the plurality of candidate shape matches each comprises an arrangement of one of the acquired determined three-dimensional room shapes of one of the plurality of rooms relative to the estimated three-dimensional room shape and has a determined degree of match based at least in part on a fit of the estimated three-dimensional room shape to the arrangement of one acquired determined three-dimensional room shape; locating, within the one room in which the acquired determined three-dimensional room shape is used for the selected one candidate shape matching, the acquisition position of the target image based on the arrangement of the selected one candidate shape matching and by using information about the target image within the estimated three-dimensional room shape; and Information is provided about the determined acquisition position of the target image in the one room of the building in which the acquired determined three-dimensional room shape is used for matching the selected one candidate shape.

2. The system according to claim 1, wherein: Determining the acquisition location of the target image in the building includes determining an orientation of the target image at the determined acquisition location, the orientation identifying a direction in at least two dimensions of a starting view from at least the determined acquisition location, and wherein providing the information includes presenting a display of at least some of the building on a client device, the display including the one room and overlaying a visual indication of the determined acquisition location and a visual indication of the determined orientation.

3. The system according to claim 1, wherein: For each of the one or more rooms, obtaining the determined room shape of the one or more rooms comprises: acquiring at least one additional panoramic image captured in the room; and The determined room shape is generated for the room based at least in part on analyzing visual content of the at least one additional panoramic image acquired in the room.

4. A computer-implemented method comprising: receiving, by one or more computing systems, a target image acquired in a building having a plurality of rooms, wherein the target image is a panoramic image having a visual overlay of a plurality of surrounding walls; Determining, by the one or more computing systems, a capture location of the target image on the floor plan of the building, comprising: obtaining, by the one or more computing systems, determined three-dimensional room shapes of the plurality of rooms; generating, by the one or more computing systems and based at least in part on analyzing visual content of the target image, an estimated three-dimensional room shape visible in the target image and an indicated location of the target image within the estimated three-dimensional room shape; Determining, by the one or more computing systems, a plurality of candidate shape matches for the estimated three-dimensional room shape, comprising: selecting, by the one or more computing systems, one or more of the determined three-dimensional room shapes as a candidate room shape for matching with the estimated three-dimensional room shape; and by the one or more computing systems, and for each candidate room shape, for each candidate shape match of one or more of the plurality of candidate shape matches, by matching one or more corners of the candidate room shape with one or more corners of the estimated three-dimensional room shape, and aligning one or more walls of the candidate room shape with one or more walls of the estimated three-dimensional room shape by using vanishing line angle information of the target image, and by adjusting a size of at least one of the estimated three-dimensional room shape or the candidate room shape to reflect each other, determining an arrangement of the candidate room shape relative to the estimated three-dimensional room shape to generate one or more candidate shape matches of the plurality of candidate shape matches; selecting, by the one or more computing systems, one of a plurality of candidate shape matches for the estimated three-dimensional room shape based at least in part on the determined degree of match of the selected one candidate shape match, wherein the plurality of candidate shape matches each include an arrangement of one of the determined three-dimensional room shapes relative to the estimated three-dimensional room shape and have a determined degree of match based at least in part on a fit of the one determined three-dimensional room shape to the arrangement of the estimated three-dimensional room shape, and wherein the one determined three-dimensional room shape of the selected one candidate shape match is an identified room of the plurality of rooms; locating, by the one or more computing systems, the indicated position with respect to the arrangement within the estimated three-dimensional room shape using the determined three-dimensional room shape relative to the selected one candidate shape matching the estimated three-dimensional room shape, and identifying the acquisition position of the target image on the floor plan based on the located indicated position within the identified room; and Information regarding the determined acquisition location of the target image in the identified room is provided by the one or more computing systems to enable display of the determined acquisition location on the floor plan of the building.

5. The computer-implemented method of claim 4, wherein: Determining the acquisition location of the target image on the floor plan includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location, the orientation identifying a direction in at least two dimensions of a starting view from at least the determined acquisition location, and wherein providing the information includes presenting, by the one or more computing systems and on a client device, a display of at least some of the floor plan of the building, the display including a visual indication of the identified room and overlaying a visual indication of the determined acquisition location and a visual indication of the determined orientation.

6. The computer-implemented method of claim 4, wherein: Generating the estimated three-dimensional room shape comprises: determining, by the one or more computing systems, room shape information of surrounding rooms surrounding the acquisition location of the target image, the room shape information comprising at least one of a three-dimensional point cloud modeling at least some of the walls of the surrounding room, or a planar surface representing at least some of the walls of the surrounding room, or a wireframe structural line representing a boundary of at least some of the walls of the surrounding room, and further comprising structural wall elements of the surrounding room, the structural wall elements comprising at least windows, doorways, and non-doorway wall openings, and The determined room shape information using the estimated three-dimensional room shape.

7. The computer-implemented method of claim 6, wherein: Generating the estimated three-dimensional room shape includes using information about the acquisition time of the target image, and also includes determining, by the one or more computing systems, additional room shape information, the additional room shape information comprising at least one of the following: a three-dimensional bounding box around the structural wall elements, or at least one room type label for the surrounding rooms, or at least one object type label for each of the structural wall elements, or at least one embedding vector generated by a trained neural network for the target image to encode information about characteristics of the surrounding rooms.

8. The computer-implemented method of claim 4, wherein: The selecting the one or more determined three-dimensional room shapes includes using information about an acquisition time of the target image, and further includes identifying, by the one or more computing systems, a subset of the determined three-dimensional room shapes for use as the selected candidate room shapes, including using at least one of: a room type label of a match between the estimated three-dimensional room shape and the selected candidate room shape; or one or more determined three-dimensional room shapes of one or more rooms of the building associated with one or more additional images acquired before and / or after the target image in the building; or one or more determined three-dimensional room shapes of one or more rooms of the building associated with one or more additional images having features that match features of the target image above a first defined matching threshold; or one or more determined three-dimensional room shapes of one or more rooms of the building associated with one or more additional images having an overall similarity to the target image by matching a common visible area above a second defined matching threshold; or One or more determined three-dimensional room shapes of one or more rooms of the building associated with one or more additional images, wherein a plurality of determined directions between each pair of the target image and one of the additional images matches above a third defined matching threshold, wherein one of the plurality of determined directions between each pair is based on visual content of the pair of images, and another one of the plurality of determined directions between each pair is based at least in part on acquisition metadata from acquisition of the target image.

9. The computer-implemented method of claim 4, wherein: The selecting one of the plurality of candidate shape matches for the estimated three-dimensional room shape based at least in part on the determined degree of match for the selected one candidate shape match comprises: determining, by the one or more computing systems and for the selected one candidate shape match, a match score between the estimated three-dimensional room shape of the selected one candidate shape match and the determined three-dimensional room shape in the arrangement of the selected one candidate shape match based on one or more matching criteria, wherein the one or more matching criteria comprise at least one of the following: a determined distance between matching corners of the estimated three-dimensional room shape and the determined three-dimensional room shape in the arrangement; or a determined distance between matching structural wall locations of said estimated three-dimensional room shape and said determined three-dimensional room shape in said arrangement; or a determined distance between matching wireframe structure lines of the estimated three-dimensional room shape and the determined three-dimensional room shape in the arrangement; or a determined distance between matched three-dimensional bounding boxes of structural wall elements in said estimated three-dimensional room shape and said determined three-dimensional room shape in said arrangement; or A determined distance between the estimated three-dimensional room shape and a matched three-dimensional bounding box of a floor element in the determined three-dimensional room shape in the arrangement.

10. A non-transitory computer-readable medium having stored content that causes one or more computing systems to perform automated operations, the automated operations comprising at least: receiving, by the one or more computing systems, a target image and a plurality of additional images acquired in a building having a plurality of rooms, wherein each of the target image and the plurality of additional images is a panoramic image having a visual overlay of a plurality of surrounding walls; Determining, by the one or more computing systems, a capture location of the target image in the building, comprising: acquiring, by the one or more computing systems, three-dimensional room shapes determined from the plurality of additional images, wherein for an associated one of the plurality of additional images, each determined three-dimensional room shape identifies at least walls and corners visible in a surrounding room; generating, by the one or more computing systems and based at least in part on analyzing visual content of the target image, an estimated three-dimensional room shape and an indicated location of the target image within the estimated three-dimensional room shape, the estimated three-dimensional room shape including at least walls and corners visible in the target image; Determining, by the one or more computing systems, a plurality of candidate shape matches for the estimated three-dimensional room shape, comprising: selecting, by the one or more computing systems, one or more of the determined three-dimensional room shapes as a candidate room shape for matching with the estimated three-dimensional room shape; and by the one or more computing systems, and for each candidate room shape, for each candidate shape match of one or more of the plurality of candidate shape matches, by matching one or more corners of the candidate room shape with one or more corners of the estimated three-dimensional room shape, and aligning one or more walls of the candidate room shape with one or more walls of the estimated three-dimensional room shape by using vanishing line angle information of the target image, and by adjusting a size of at least one of the estimated three-dimensional room shape or the candidate room shape to reflect each other, determining an arrangement of the candidate room shape relative to the estimated three-dimensional room shape to generate one or more candidate shape matches of the plurality of candidate shape matches; selecting, by the one or more computing systems, one of a plurality of candidate shape matches for the estimated three-dimensional room shape based at least in part on a degree of match determined for the selected one candidate shape match, wherein the plurality of candidate shape matches each include an arrangement of one of the determined three-dimensional room shapes relative to the estimated three-dimensional room shape and have a determined degree of match based at least in part on a fit of the walls and corners of the estimated three-dimensional room shape using the candidate shape match with the arrangement of the walls and corners of the one determined three-dimensional room shape of the candidate shape match; and, by the one or more computing systems and within the surrounding room, for the additional images associated with the one determined three-dimensional room shape that matches the selected one candidate shape, locating the indicated position within the estimated three-dimensional room shape using the arrangement of the selected one candidate shape matches, and identifying the acquisition position of the target image within the building based on the located indicated position within the surrounding room; Information about the determined acquisition location of the target image within the surrounding room is provided by the one or more computing systems to enable the determined acquisition location to be displayed using information about the building.

11. The non-transitory computer readable medium of claim 10, wherein: Determining the acquisition location of the target image in the building includes determining, by the one or more computing systems, an orientation of the target image at the determined acquisition location, the orientation identifying a direction in at least two dimensions of a starting view from at least the determined acquisition location, and wherein providing the information includes presenting, by the one or more computing systems and on a client device, a display of at least some of the building, the display including a visual indication of the surrounding rooms and overlaid with a visual indication of the determined acquisition location and a visual indication of the determined orientation.

12. The non-transitory computer readable medium of claim 10, wherein: The obtaining of the determined three-dimensional room shape comprises, for each of one or more rooms in the building, generating the determined three-dimensional room shape of the room by analyzing visual content of at least one of the plurality of additional images acquired in the room, wherein the determination of the acquisition location of the target image is performed dynamically when the target image is acquired, and wherein the providing of the information comprises presenting, by the one or more computing systems and when the target image is acquired, a display of at least some of the building including the surrounding rooms.

13. The non-transitory computer readable medium of claim 10, wherein: Selecting one of the plurality of candidate shape matches for the estimated three-dimensional room shape based at least in part on the determined degree of match of the selected one candidate shape match comprises: determining, by the one or more computing systems and for the selected one candidate shape match, a match score between the estimated three-dimensional room shape of the selected one candidate shape match and the determined three-dimensional room shape in the arrangement of the selected one candidate shape match based on one or more matching criteria, wherein the one or more matching criteria comprises at least one of: a distance between matching corners of the estimated three-dimensional room shape and the determined three-dimensional room shape in the arrangement; or a distance between matching structural walls of the estimated three-dimensional room shape and the determined three-dimensional room shape in the arrangement; or a distance between matching wireframe structure lines of the estimated three-dimensional room shape and the determined three-dimensional room shape in the arrangement; or a distance between matching three-dimensional bounding boxes of structural wall elements in said estimated three-dimensional room shape and said determined three-dimensional room shape in said arrangement; or a distance between matching three-dimensional bounding boxes of floor elements in the estimated three-dimensional room shape and the determined three-dimensional room shape in the arrangement; or for said selected one candidate shape matching said determined three-dimensional room shape, a difference between the visual content of said target image and the visual content of an additional image associated with said surrounding room; or A difference between two measurements of direction between the target image and the one additional image, the two measurements comprising a first measurement based on the visual content of the target image and the visual content of the one additional image and a second measurement based at least in part on acquisition metadata from acquiring the target image and the one additional image.

14. The non-transitory computer readable medium of claim 10, wherein: Selecting one of the plurality of candidate shape matches for the estimated three-dimensional room shape based at least in part on a determined degree of match for a selected one candidate shape match includes determining, by the one or more computing systems, the degree of match for the selected one candidate shape match based at least in part on comparing a first generated image embedding vector of the target image and a second generated image embedding vector of the one additional image associated with the surrounding room of the determined three-dimensional room shape that matches the selected one candidate shape, wherein the first generated image embedding vector encodes information about one or more features and / or attributes of the target image, and wherein the second generated image embedding vector encodes information about one or more features and / or attributes of the one additional image.