Automated analysis of image content to determine image acquisition locations

Abstract

Techniques are described for using a computing device to perform automated operations for determining an acquisition location of an image using analysis of the visual content of the image. In at least some cases, the image to be analyzed includes a panoramic image acquired at an acquisition location in the interior of a multi-room building, and the determined acquisition location information includes a location on a floor plan of the building and, in some embodiments, directional direction information—in at least some such cases, the acquisition location determination is performed without having or using information from any distance measuring device regarding distances from the image acquisition location to objects in the surrounding building. The acquisition location information can be used in a variety of automated ways, including for controlling navigation of a device (e.g., an autonomous vehicle), for display in a corresponding graphical user interface on one or more client devices, and the like.

Description

Technical Field

[0001] The following disclosure generally relates to techniques for automatically determining the acquisition location of an image based on analysis of the image content and subsequently using the determined acquisition location information in one or more ways, such as locating interior images of rooms in a building on a floor plan of the building and using the image location to improve navigation of the building. Background Art

[0002] In various fields and situations, such as building analysis, property inspections, real estate acquisition and development, general contracting, cost estimate improvement, etc., it may be desirable to know the interior of a house, office, or other building without having to physically travel to and enter the building. However, it may be difficult to effectively capture, represent, and use information about the interior of such a building, including displaying visual information captured within the interior of the building to a user at a remote location (e.g., enabling the user to fully understand the layout and other interior details, including controlling the display in a manner selected by the user). In addition, although a floor plan of a building may provide some information about the layout and other details of the interior of the building, such use of a floor plan has some disadvantages, including that the floor plan may be difficult to construct and maintain, difficult to accurately scale and populate with information about the interior of a room, difficult to visualize and otherwise use, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1A and Figure 1B is a diagram depicting an exemplary building interior environment and computing system used in embodiments of the present disclosure, including generating and presenting information representing the building interior.

[0004] Figures 2A to 2G An example is shown of automatically generating and presenting information about a building floor plan based on one or more images of the interior of the building, such as automatically determining and presenting the acquisition locations of the images based on analysis of the image content.

[0005] Figure 3 is a block diagram illustrating a computing system suitable for executing an embodiment of a system that performs at least some of the techniques described in this disclosure.

[0006] Figure 4 An exemplary implementation of a flow chart for an image capture and analysis (ICA) system routine is shown in accordance with an embodiment of the present disclosure.

[0007] Figure 5A and Figure 5B An exemplary embodiment of a flow chart for a Mapping Information Generation Manager (MIGM) system routine is shown in accordance with an embodiment of the present disclosure.

[0008] Figure 6An exemplary implementation of a flow chart for an Image Location Mapping Manager (ILMM) system routine is shown in accordance with an implementation of the present disclosure.

[0009] Figure 7 An exemplary implementation of a flow chart for a building map viewer system routine is shown in accordance with an implementation of the present disclosure. DETAILED DESCRIPTION

[0010] The present disclosure describes techniques for using a computing device to perform automated operations related to determining the acquisition location of an image based at least in part on an analysis of the image content (e.g., visual information present in the image) and for subsequently using the determined image acquisition location information in one or more further automated ways. In at least some embodiments, the images to be analyzed include one or more panoramic images or other images (e.g., linear perspective images) acquired at one or more acquisition locations inside a multi-room building (e.g., a house, an office, etc.), and the determined image acquisition location information includes at least a location on a floor plan of the building, and in some cases further includes an orientation or other direction information for at least a portion of the image. In at least some such embodiments, the automatic image acquisition location determination is further performed without having or using information from any depth sensor or other distance measuring device about the distance from the image acquisition location to walls or other objects in the surrounding building. In various embodiments, the determined image acquisition location information can be further used in various ways, such as in conjunction with a corresponding building floor plan and / or other mapping information generated, including for controlling navigation of a mobile device (e.g., an autonomous vehicle), for display or other presentation on one or more client devices in a corresponding GUI (graphical user interface), etc. Additional details regarding the automatic acquisition and use of determined image acquisition location information are included below, and as further discussed below, in at least some embodiments, some or all of the techniques described herein may be performed via automated operation of an Image Location Mapping Manager (“ILMM”) system.

[0011] In at least some embodiments and scenarios, some or all of the images collected for the building may be panoramic images collected at one of a plurality of collection locations in or around the building, respectively, such as generating a panoramic image at each such collection location based on one or more of: one or more videos at the collection location (e.g., 360° video captured by a smartphone or other mobile device held by a user turning around at the collection location), or multiple images collected from the collection location in multiple directions (e.g., from a smartphone or other mobile device held by a user turning around at the collection location), or simultaneous capture of all image information (e.g., using one or more fisheye lenses), etc. It will be appreciated that in some cases, such panoramic images may be represented in a spherical coordinate system and provide up to 360° coverage around horizontal and / or vertical axes, so that a user viewing a starting panoramic image may move the viewing direction within the starting panoramic image to different orientations to produce different images (or “views”) to be rendered within the starting panoramic image (including converting the image being rendered to a planar coordinate system if the panoramic image is represented in a spherical coordinate system). Additionally, acquisition metadata regarding the capture of such panoramic images may be obtained and used in various ways, such as data collected from an IMU (inertial measurement unit) sensor or other sensor of a mobile device as the mobile device is carried by a user or otherwise moved between acquisition locations. Additional details regarding the acquisition and use of panoramic or other images of a building are included below.

[0012] As described above, automated operation of the ILMM system may include determining the acquisition location of an image captured in a defined area (e.g., within a room of a house or other building) based, at least in part, on an analysis of visual information included in the image content. In at least some embodiments, such automatic determination of the image acquisition location may include some or all of the following: identifying visible elements in the image content, such as structural elements of a visible wall (e.g., wall elements such as doors, windows, boundaries between walls, etc.); determining the locations of these elements identified within the visual information of the image content at various angles; encoding the identified angular element location information in a format that facilitates comparison with similar information for one or more rooms of the building (the resulting encoded angular element location information of the image is generally referred to herein as an image angular descriptor); and using the generated image angular descriptor to identify matching information for a particular location in a particular room. In other embodiments and scenarios, the wall element information (or other types of visible elements) of the image may be identified and represented in a manner other than based on an angular difference from the starting direction of the image, thereby generating other types of image descriptors used in a similar manner.

[0013] For purposes of an illustrative example, consider a panoramic image captured in a room of a building, wherein the panoramic image includes 360° horizontal coverage around a vertical axis (e.g., showing a full circle of all room walls from the acquisition location of the panoramic image), and wherein the x and y axes of the visual content of the image are aligned with corresponding horizontal and vertical information in the room (e.g., a boundary between two walls, a boundary between a wall and a floor, the bottom and / or top of windows and doors, etc.) such that the image is not tilted or otherwise misaligned relative to the room. For purposes of this example, image capture may be performed sequentially from the acquisition location in multiple directions using a changing camera orientation starting with a camera orientation corresponding to a relative starting horizontal direction of 0° north for the panoramic image and continuing in a circular manner, wherein the relative 90° horizontal direction of the panoramic image then corresponds to east, the relative 180° horizontal direction of the panoramic image corresponds to south, the relative 270° horizontal direction of the panoramic image corresponds to west, and the relative 360° ending horizontal direction of the panoramic image returns to north. In at least some embodiments, information about the position of identified elements in a panoramic image is encoded in a manner that is specific to such an orientation angle number from the acquisition location (e.g., relative to the starting direction of the panoramic image), thereby producing an image angle descriptor for the image—thus, an image angle descriptor for such an image can encode information about which wall elements are visible in each of the 360 ​​horizontal degrees. For example, if a window exists in a room that is located due north of the image acquisition location and is visible in the panoramic image (e.g., not obscured by intervening furniture), then information about the 0° relative orientation angle of the image in the resulting image angle descriptor (and, if represented separately from the 0° orientation, about the 360° relative orientation of the image) will include an identification of the presence of the window in that angular orientation. In various embodiments, such positional information about identified elements can be encoded and stored in various manners, including in some embodiments in a vector having one or more values ​​for each orientation angle, such as identifying each wall element that exists in a given angular orientation. Additional details regarding the construction and use of such image angle descriptors are included below, including information regarding FIG. 2D to FIG. 2E Examples and their associated descriptions.

[0014] Additionally, using the generated image angle descriptors for an image to automatically determine the acquisition location of the image in a room (or other defined area) of a building may include matching such angle information for the image with corresponding angle information in the building, such as with a particular location in the room. In at least some embodiments, a plurality of room locations are identified in the building (e.g., by creating a grid of room locations that substantially or completely covers the floor of the room), and a building angle descriptor is created for each such room location to include similar angle information about the room wall elements at that room location—thus, given a specified starting direction of 0° (e.g., north). Such a building angle descriptor for a room location may encode information about which wall elements are part of the room for each of the 360 ​​horizontal degrees in a direction from the room location. Such a building angle descriptor may be predefined, for example, prior to generating or using any corresponding image angle descriptors, or may instead be dynamically created at the time of use in some cases for comparison with image angle descriptors for images captured in the room.

[0015] Once a plurality of building corner descriptors are generated or otherwise obtained for a plurality of room locations in a room, they may be compared or otherwise matched with image corner descriptors of images captured in the room to determine the best matching one of the building corner descriptors, and then identify the acquisition location of the image based on the room location of the best matching building corner descriptor. For example, the determined acquisition location of the image may be selected as the room location that best matches the building corner descriptor in some embodiments and scenarios, or may instead be determined to be close to the room location (e.g., in a direction and / or amount based on the difference between the image corner descriptor and the best matching building corner descriptor) in other embodiments and scenarios. The matching process for the image corner descriptors and the building corner descriptors may include determining the distance and / or amount of similarity / dissimilarity between the two corner descriptors in one or more ways, such as by determining the probability that the two corner descriptors match (with the highest probability of matching corresponding to the smallest dissimilarity and / or distance), by measuring the difference between vectors or other encoding formats of the corner descriptors used for comparison, etc. As a non-exclusive example, a circular bulldozer distance metric can be used to compare vectors of two such corner descriptors in a rotation-independent manner (e.g., regardless of whether the two corner descriptors use the same orientation as their respective relative 0° in the room), while in other embodiments, rotational differences between the two descriptors can be handled in other ways. Additionally, in some embodiments, the matching process can include comparing the image corner descriptor to every possible building corner descriptor, while in other embodiments, only a subset of the building corner descriptors can be considered (e.g., by performing a nearest neighbor gradient ascent or descent search using a defined similarity or dissimilarity metric). Additional details regarding the construction and use of such building corner descriptors are included below, including comparing one or more image corner descriptors, such as with respect to FIG. Figure 2E Examples and their associated descriptions.

[0016] Additionally, in at least some embodiments, further automated operations may be performed as part of the automatic determination of the acquisition location of the image acquired in the room. For example, in at least some embodiments, geometric positioning techniques may be used to test the association of wall elements visible in the image with wall elements present in the room (whether to confirm the degree of match of building corner descriptors that have been determined to be the best match with the image corner descriptors and / or as part of identifying such best matching building corner descriptors). The geometric positioning techniques may include, for example, using a 2-point solver and / or a 3-point solver to determine one or more possible room shapes for the room and / or element locations within the room, and then positioning the wall elements on the possible room shapes. In other embodiments, wall element locations may be determined in other ways, such as via the use of depth sensing devices or other room mapping sensors in the room. By machine learning methods for analyzing images to identify room shapes and wall element locations, by inputs specified by one or more operators, and the like. Additionally, in some embodiments, given a room location and information about the room shape and location of wall elements, a new synthetic image can be generated with the wall elements shown in their locations, the new synthetic image being a projection / visualization of a view of the room from that room location, and the visual information of the synthetic image can be compared directly with the actual image from the room to determine a degree of similarity / dissimilarity or other degree of match between the two images. This inter-image comparison is used to determine whether the room location matches the acquisition location of the actual image. In a similar manner, in some embodiments, some or all of the building angle descriptors for room locations in the room can be generated as image angle descriptors of images (e.g., 360° panoramic images) taken at those room locations, and those room / image angle descriptors can then be compared with image angle descriptors of a new image (e.g., an image with less than 360° horizontal coverage) taken in the room to determine the best matching building angle descriptors in a manner similar to that discussed above.

[0017] In some embodiments, automated determination of the acquisition location of an image captured in a room can further include additional operations. For example, in at least some embodiments, machine learning techniques can be used to learn the best encoding to allow the image to match the room location, such as from a plurality of defined candidate encodings, or instead by considering a plurality of possible image elements to analyze and identify a subset of those image elements that provide the best match to the corresponding room location. The following includes additional details on various automated operations that can be performed by the ILMM system in some embodiments.

[0018] In various embodiments, the described techniques provide various benefits, including allowing floor plans of multi-room buildings and other structures to be automatically augmented with information about acquisition locations at which images were acquired in the building or other structure, including without having to have or using information from a depth sensor or other distance measuring device about the distance from the image acquisition location to walls or other objects in the surrounding building or other structure. In addition, such automated techniques allow such image acquisition location information to be determined more quickly than previously existing techniques, and with greater accuracy in at least some embodiments, including by using information acquired from the actual building environment (rather than from plans about how the building should theoretically be built), and enabling the capture of changes to structural elements that occur after the building was initially constructed. Such described techniques further provide benefits in allowing improved automated navigation of buildings by mobile devices (e.g., semi-autonomous or fully autonomous vehicles) based at least in part on the determined image acquisition locations, including significantly reducing the computing power and time used to attempt to otherwise learn the layout of the building. Additionally, in some implementations, the described techniques can be used to provide an improved GUI in which a user can more accurately and quickly obtain information about a building's interior (e.g., for use in navigating the interior), including in response to a search request, as part of providing personalized information to a user, as part of providing a user with value estimates and / or other information about a building, etc. The described techniques also provide a variety of other benefits, some of which are further described elsewhere herein.

[0019] As described above, the automated operation of the ILMM system may include determining the acquisition location of an image captured in a defined area (e.g., in a room of a house or other building) based, at least in part, on an analysis of the visual information included in the image content. In at least some embodiments, such an ILMM system may be operated in conjunction with one or more separate ICA (image capture and analysis) systems and / or one or more separate MIGM (mapping information and generation manager) systems, such as obtaining and using building floor plans and other associated information from ICA and / or MIGM systems, while in other embodiments, such an ILMM system may incorporate some or all of the functions of such ICA and / or MIGM systems as part of the ILMM system. In yet other embodiments, the ILMM system may operate without using some or all of the functions of the ICA and / or MIGM systems, such as if the ILMM system obtains information about building floor plans and / or other associated information from other sources (e.g., from manual creation by one or more users, from provision of such building floor plans and / or associated information by one or more external systems or other sources, etc.). Additionally, building floor plans used in the manner described herein may be in a variety of formats (whether initially obtained and / or following initial automated analysis by the ILMM system), including in at least some embodiments in a vectorized form having specified information regarding the location of structural elements, such as one or more of: walls, windows, doorways and other openings between rooms, corners, etc. (e.g., after initially receiving a building floor plan in a non-vectorized image form that is analyzed to produce a vectorized form).

[0020] With respect to the functionality of such an ICA system, in at least some embodiments, it may perform automated operations to capture images (e.g., panoramic images) at various capture locations associated with a building (e.g., inside multiple rooms of a building), and optionally further capture metadata related to the image capture process and / or the movement of the capture device between the capture locations. For example, in at least some such embodiments, such techniques may include: using one or more mobile devices (e.g., a camera having one or more fisheye lenses and mounted on a rotatable tripod or otherwise having an automated rotation mechanism; a camera having one or more fisheye lenses sufficient to capture 360 ​​degrees horizontally without rotation; a smartphone held and moved by a user, such as rotating the user's body in a 360° circle around a vertical axis and holding the smartphone; a camera held by a user or mounted on the user's body or on the user's clothing; a camera mounted on an aerial and / or ground-based drone or robotic device; etc.) to capture visual data from a series of multiple capture locations in multiple rooms of a house (or other building), but without the collection of information from any depth sensor or other distance measurement device regarding the distance between the capture locations and objects in the environment surrounding the capture locations. Additional details regarding the operation of an apparatus implementing an ICA system are included elsewhere herein, such as performing such automated operations and, in some cases, further interacting in one or more ways with one or more ICA system operator users to provide further functionality.

[0021] With respect to the functionality of such a MIGM system, in at least some embodiments, it may perform automated operations to analyze a plurality of 360° panoramic images (and optionally other images) that have been acquired of a building interior (and optionally a building exterior) and determine, for some or all of those panoramic images, the locations of room shapes and passageways connecting the rooms, and in at least some embodiments and scenarios, wall elements and other elements of some or all of the rooms of the building. The types of connecting passageways between two or more rooms may include one or more of: doorway openings and other inter-room non-doorway wall openings, windows, stairs, non-room corridors, etc., and the automated analysis of the images may identify such elements based at least in part on identifying the outlines of passageways, identifying content within passageways that is different from that outside of them (e.g., different colors or shading), etc. The automated operations may further include: using the determined information to generate a floor plan of the building and optionally generating other mapping information of the building (such as by using the inter-room passageway information and other information to determine the relative locations of associated room shapes to each other), and optionally adding distance scaling information and / or various other types of information to the generated floor plan. Additionally, in at least some embodiments, the MIGM system can perform further automated operations to determine and associate additional information with a building floor plan and / or a particular room or location within a floor plan, such as analyzing images and / or other environmental information (e.g., audio) captured within the building to determine specific attributes (e.g., the color and / or material type and / or other characteristics of a particular element, such as a floor, wall, ceiling, countertop, furniture, light fixture, appliance, etc.; the presence and / or absence of a particular element, such as an island in a kitchen, etc.), or otherwise determine relevant attributes (e.g., the direction a building element faces, such as a window; a view from a particular window or other location; etc.). The following includes additional details regarding the operation of a computing device implementing the MIGM system, such as performing such automated operations, and in some cases further interacting with one or more MIGM system operator users in one or more ways to provide further functionality.

[0022] For illustrative purposes, some embodiments are described below in which specific types of information are collected, used, and / or presented in specific ways for specific types of structures and by using specific types of devices - however, it will be understood that the described techniques can be used in other ways in other embodiments, and thus the invention is not limited to the exemplary details provided. As a non-exclusive example, although in some embodiments specific types of corner descriptors are generated for images and room locations and compared or otherwise matched in a specific manner, it will be understood that other types of information used to describe image content and room locations can be similarly generated and used in other embodiments, including for buildings (or other structures or layouts) that are separate from the house, and the determined image collection location information can be used in other ways in other embodiments. In addition, the term "building" refers to any partially or completely enclosed structure in this document, typically but not necessarily including one or more rooms that visually or otherwise divide the interior space of the structure - non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, condominiums, office buildings, commercial buildings or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.), etc. The term "collection" or "capture" as used herein with reference to a building interior, a collection location, or other location (unless the context clearly indicates otherwise) may refer to any recording, storage, or entry of media, sensor data, and / or other information related to spatial characteristics and / or visual characteristics and / or otherwise perceptible characteristics of a building interior, or a subset thereof, such as by a recording device or by another device that receives information from a recording device. As used herein, the term "panoramic image" may refer to a visual representation based on, including, or separable into a plurality of discrete component images originating from substantially similar physical locations in different directions and depicting a larger field of view than any of the discrete component images depict alone, including images from a physical location with a sufficiently wide viewing angle to include angles beyond those perceptible by a person gazing in a single direction. As used herein, the term "series" of collection locations generally refers to two or more collection locations, each of which is visited at least once in a corresponding order, regardless of whether other non-collection locations are visited between them, and regardless of whether the visits to the collection locations occur during a single continuous time period or at multiple different times, or by a single user and / or device or by multiple different users and / or devices. In addition, various details are provided in the drawings and text for illustrative purposes, but these details are not intended to limit the scope of the present invention. For example, the sizes and relative positions of elements in the drawings are not necessarily drawn to scale, and some details are omitted and / or provided more prominently (e.g., via size and positioning) to enhance readability and / or clarity. In addition, the same reference numerals may be used in the drawings to identify the same or similar elements or actions.

[0023] Figure 1A is an example block diagram of various computing devices and systems that can participate in the described techniques in some implementations. In particular, Figure 1A 1 shows one or more linked panoramic images 165 that have been generated by an internal capture and analysis ("ICA") system 160, which in this example is executed on one or more server computing systems 180, such as with respect to one or more buildings or other structures. Figure 1B An example of such a linked panoramic image of a particular house 198 is shown, as discussed further below, and additional details related to the automated operation of the ICA system are included elsewhere herein (including with respect to Figure 4 MIGM (Mapping Information Generation Manager) system 160 is further described in Figure 1A 180 to generate and provide building floor plans 155 and / or other mapping related information based on the use of linked panoramic images 165 and optionally associated metadata regarding their acquisition and linking. Figure 2G An example of such a floor plan is shown, as discussed further below, and additional details related to the automated operation of the MIGM system are included elsewhere herein (including with respect to Figure 5A-5B additional details).

[0024] Figure 1A Further shown is an ILMM (image location mapping manager) system 140 that executes on one or more server computing systems 180 to determine the acquisition location of additional images 145 (e.g., panoramic images) acquired in one or more building rooms, such as with respect to corresponding building floor plans 155. In at least some embodiments and scenarios, one or more users of the ILMM client computing devices 105 can further interact with the ILMM system 140 via the network 170, such as to assist in some automated operations of the ILMM system for determining the acquisition location of the image based at least in part on analysis of the image content, and / or for subsequently using the determined image acquisition location information in one or more further automated manners. Additional details regarding the automated operations of the ILMM system are included elsewhere herein (including with respect to Figures 2D to 2G and Figure 6). In some embodiments, the ICA system 160 and / or the MIGM system 160 and / or the ILMM system 140 may be executed on the same server computing system, such as if multiple or all of those systems are operated by a single entity or otherwise coordinated with one another (e.g., some or all of the functionality in those systems are integrated together into a larger system), while in other embodiments, the ILMM system may instead obtain floor plan information and / or additional images from one or more external sources and optionally store them locally with the ILMM system (not shown) for further analysis and use.

[0025] One or more users (not shown) of one or more client computing devices 175 may further interact with the ILMM system 140 and optionally the ICA system 160 and / or the MIGM system 160 via one or more computer networks 170, such as to assist in determining the acquisition location of one or more images and obtain corresponding determined acquisition location information, and / or obtain and optionally interact with a generated floor plan (on which one or more additional images are already located), and / or obtain and optionally interact with additional information, such as one or more associated images (e.g., changing between a floor plan view and a view of a particular image at an acquisition location within or near the floor plan; changing the horizontal and / or vertical viewing direction according to which a corresponding view of a panoramic image is displayed, such as determining the portion of the panoramic image to which the current user viewing direction is directed, etc.). Additionally, although in Figure 1A Although not shown in FIG. 1 , a floor plan (or portion thereof) may be linked or otherwise associated with one or more other types of information, including enabling a floor plan of a multi-story or otherwise multi-story building to have multiple associated sub-floor plans for different floors or layers that are linked to each other (e.g., via connecting stairways), enabling a two-dimensional (“2D”) floor plan of a building to be linked or otherwise associated with a three-dimensional (“3D”) representation of the building, etc. Additionally, although in FIG. Figure 1A Not shown, but in some embodiments, the client computing device 175 (or other device, not shown) may receive and use the determined image acquisition location information (optionally in combination with the generated floor plan and / or other generated mapping-related information) in additional ways, such as to control or assist automated navigation activities performed by those devices (e.g., by an autonomous vehicle or other device), whether in lieu of or in addition to displaying the generated information.

[0026] exist Figure 1AIn the depicted computing environment, network 170 may be one or more publicly accessible (possibly operated by a variety of different parties) linked networks, such as the Internet, while in other embodiments, network 170 may have other forms (e.g., private networks, such as corporate or university networks that are fully or partially inaccessible to non-privileged users). In still other embodiments, network 170 may include both private and public networks, wherein one or more of the private networks may access one or more of the public networks and / or access one or more of the private networks from one or more of the public networks. Furthermore, in various scenarios, network 170 may include various types of wired and / or wireless networks. Additionally, client computing devices 175 and server computing systems 180 may include various hardware components and stored information, as described below with respect to Figure 3 discussed in more detail.

[0027] exist Figure 1AIn an example of , the ICA system 160 may perform automated operations involving generating multiple panoramic images (e.g., each being a 360 degree panorama about a vertical axis) at multiple associated acquisition locations (e.g., in multiple rooms or other locations within a building or other structure and optionally around some or all of the exterior of the building or other structure), such as for generating and providing a representation of the interior of a building or other structure. The technique may further include: analyzing information to determine a relative position / orientation between each of two or more acquisition locations; creating an inter-panorama position / orientation link in the panorama to each of one or more other panoramas based on such determined position / orientation; and then, providing information to display or otherwise present multiple linked panoramic images of the respective acquisition locations within the premises. Additional details regarding implementations of systems that provide at least some of this functionality of an ICA system are included in the following: co-pending U.S. non-provisional patent application Ser. No. 16 / 693,286 (Attorney Docket No. 990074.402C1), entitled “Connecting And Using Building Data Acquired From Mobile Devices,” filed on Nov. 23, 2019 (which includes disclosure of an example BICA system generally relating to obtaining and using panoramic images from within one or more buildings or other structures); co-pending U.S. non-provisional patent application Ser. No. 16 / 693,286 (Attorney Docket No. 990074.402C1), entitled “Connecting And Using Building Data Acquired From Mobile Devices,” filed on Dec. 28, 2018 (which includes disclosure of an example BICA system generally relating to obtaining and using panoramic images from within one or more buildings or other structures); and U.S. Non-Provisional Patent Application No. 16 / 236,187 (Agent Docket No. 990074.405), entitled “Automated Mapping Information Generation From Inter-Connected Images,” filed on November 14, 2018; each of the foregoing patent applications is incorporated herein by reference in its entirety.

[0028] Figure 1BA block diagram of an exemplary building interior environment is depicted in which a linked panoramic image has been generated and is ready for use in generating and providing a corresponding building floor plan and for presenting the linked panoramic image to a user. In particular, Figure 1B A building 198 is included that has an interior that is at least partially captured via a plurality of panoramic images, such as by a user (not shown) carrying a mobile device 185 with image capture capabilities traversing the building interior to a series of plurality of capture locations 210. An embodiment of an ICA system (e.g., an ICA system 160 on a server computing system 180; a copy of some or all of the ICA system executing on a user's mobile device, such as an ICA application system 155 executing in memory 152 of device 185; etc.) may automatically perform or assist in capturing data representing the building interior, and further analyzing the captured data to generate linked panoramic images to provide a visual representation of the building interior. Although the user's mobile device may include various hardware components, such as a camera or other imaging system 135, one or more sensors 148 (e.g., gyroscopes 148a, accelerometers 148b, compasses 148c, etc., such as part of one or more IMUs (or inertial measurement units) of the mobile device; altimeters; light detectors; etc.), a GPS receiver, one or more hardware processors 132, memory 152, display 142, microphones, etc., in at least some embodiments, the mobile device may not have access to or use equipment to measure the depth of objects in the building relative to the location of the mobile device, so that the relationship between different panoramic images and their acquisition locations can be determined based in part or in whole on matching elements in different images and / or by using information from other hardware components listed (but not using any data from any such depth sensor). In addition, although a direction indicator 109 is provided for viewer reference, in at least some embodiments, the mobile device and / or ICA system may not use such absolute direction information, such as instead determining the relative direction and distance between the panoramic images 210 without regard to actual geographic location or direction.

[0029] In operation, a user associated with a mobile device arrives at a first acquisition location 210A within a first room within a building (in this example, the entrance to the living room from the exterior door 190-1), and as the mobile device is rotated about a vertical axis at the first acquisition location (e.g., where the user turns his or her body in a circle while keeping the mobile device fixed relative to the user's body), a view of a portion of the building interior visible from the acquisition location 210A (e.g., some or all of the first room, and optionally a small portion of one or more other adjacent or nearby rooms, such as through a door, hall, staircase, or other connecting passageway from the first room) is captured. The actions of the user and / or mobile device may be controlled or facilitated through the use of one or more programs executed on the mobile device, such as the ICA application system 155, the optional browser 162, the control system 147, etc., and the view capture may be performed by recording a video and / or taking a sequence of one or more images, including capturing visual information depicting a number of objects or other elements (e.g., structural details) that may be visible in an image (e.g., a video frame) captured from the acquisition location. Figure 1B In the example of , such objects or other elements include various elements that are structurally part of the wall (or "wall elements"), such as doorways 190 and 197 and their doors (e.g., with revolving doors and / or sliding doors), windows 196, wall boundaries (e.g., corners or edges) 195 (including corner 195-1 at the northwest corner of the building 198, and corner 195-2 at the northeast corner of the first room) - in addition, in Figure 1B In the example of , such objects or other elements may further include other elements within the room, such as furniture 191-193 (e.g., sofa 191; chair 192; table 193; etc.), pictures or paintings hanging on the wall or televisions or other objects 194 (such as, 194-1 and 194-2), lighting fixtures, etc. The user may also optionally provide a text or auditory identifier to be associated with the collection location, such as "entrance" for collection location 210A or "living room" for collection location 210B, while in other embodiments, the ICA system may automatically generate such an identifier (e.g., by automatically analyzing video and / or other recorded information of the building to perform a corresponding automated determination, such as by using machine learning), or may not use an identifier.

[0030] After the first acquisition location 210A has been appropriately captured (e.g., by a full rotation of the mobile device), the user can proceed to the next acquisition location (such as acquisition location 210B), optionally recording movement data, such as video and / or other data from hardware components (e.g., from one or more IMUs, from cameras, etc.) during movement between acquisition locations. At the next acquisition location, the user can similarly use the mobile device to capture one or more images from that acquisition location. This process can be repeated from some or all of the rooms of the building, and optionally repeated outside the building, as shown for acquisition locations 210C to 210J. The video and / or other images acquired for each acquisition location are further analyzed to generate a panoramic image for each of the acquisition locations 210A to 210J, including, in some embodiments, matching objects and other elements in different images. In addition to generating such a panoramic image, further analysis may be performed to 'link' at least some of the panoramas together (for illustration, some corresponding lines 215 between them are shown), such as determining relative position information between multiple pairs of acquisition locations that are visible to each other, storing corresponding inter-panoramic links (e.g., link 215-AB between acquisition locations A and B, link 215-BC between B and C, and link 215-AC between A and C, respectively), and in some embodiments and scenarios, further linking at least some acquisition locations that are not visible to each other (e.g., link 215-BE between acquisition locations 210B and 210E, not shown).

[0031] Additional details regarding embodiments of generating and using linking information between panoramic images (including using path of travel information and / or elements or other features visible in multiple images) are included in co-pending U.S. non-provisional patent application Ser. No. 16 / 693,286 (Agent Docket No. 990074.402C1), filed on Nov. 23, 2019, and entitled “Connecting And Using Building Data Acquired From Mobile Devices,” which includes disclosure of an example BICA system generally relating to obtaining and using linking information to interconnect multiple panoramic images captured within one or more buildings or other structures.

[0032] about Figure 1A to Figure 1B Various details are provided, but it will be understood that the details provided are non-exclusive examples included for illustrative purposes and that other implementations may be performed in other ways without some or all of such details.

[0033] Figures 2A to 2GThe invention shows a method for automatically generating and presenting information about a building (such as, for example, Figure 1B An example of information about a floor plan of a building 198).

[0034] In particular, Figure 2A An example image 250a is shown, such as in Figure 1B 190a) of a living room of a house 198 taken from acquisition location 210B in a northeast direction (or a northeast-facing subset view of a 360-degree panoramic image taken from the acquisition location and formatted in a rectilinear manner)—in this example, a direction indicator 109a is further displayed to show the northeast direction in which the image was taken). In the example shown, the displayed image includes built-in elements (e.g., lighting fixture 130a), furniture (e.g., chair 192-1), two windows 196-1, and a picture 194-1 hanging on the north wall of the living room. Inter-room passages (e.g., doors or other wall openings) leading into and out of the living room are not visible in this image. However, multiple room boundaries are visible in image 250a, including a horizontal boundary between a visible portion of the north wall of the living room and the living room ceiling and floor, a horizontal boundary between a visible portion of the east wall of the living room and the living room ceiling and floor, and an inter-wall vertical boundary 195-2 between the north wall and the east wall.

[0035] Figure 2B continue Figure 2A of examples, and shows that Figure 1B 198 of the living room of the house 198 taken from the acquisition location 210B in a northwest direction - the direction indicator 109b is further displayed to show the northwest direction in which the image was taken. In this example image, a small portion of one of the windows 196-1 continues to be visible along with a portion of the window 196-2 and the new lighting fixture 130b. In addition, the horizontal and vertical room boundaries are shown with Figure 2A A similar approach is visible in image 250b.

[0036] Figure 2C continue FIG. 2A to FIG. 2B of examples, and shows that Figure 1B A third perspective image 250c is captured in the living room of house 198, such as from acquisition location 210B in a southwest direction—direction indicator 109c is further displayed to show the southwest direction in which the image was captured. In this example image, a portion of window 196-2 continues to be visible, as does sofa 191 and visual horizontal and vertical room boundaries. Figure 2A and Figure 2B This example image further shows the room passage of the living room, which in this example is the door 190-1 ( Figure 1BIt is identified as a door leading to the exterior of the house.) It will be appreciated that a variety of other perspective images may be captured and displayed in a similar manner from acquisition location 210B and / or other acquisition locations.

[0037] Figure 2D continue FIG. 2A to FIG. 2C and shows the example with Figure 1B The panoramic image 255d (where the image is being captured at the acquisition position 210) is acquired separately from the image Figure 1B 255d is an image of a living room taken from acquisition location 265, as shown using information 265 and 267 on floor plan excerpt 260d corresponding to the living room (with respect to FIG. 255d ). Figure 2G (See also the discussion of additional details regarding example displays of floor plans of building 198 elsewhere herein.) Using such panoramic image 255d, it is possible to view the panoramic image 255d in a manner similar to that of FIG. FIG. 2A to FIG. 2B Various subsets of the panoramic image are displayed to an end user (not shown) in a manner similar to the perspective images 250a-250b of the present invention, with example subset 250d being shown as a portion of panoramic image 255d—although not separately shown on panoramic image 255d, its subset portion corresponding to first perspective image 250a is available in the right portion of panoramic image 255d, while the left subset portion of panoramic image 255d contains visual data corresponding to the visual data of perspective image 250b. Since panoramic image 255d does not extend to a full horizontal 360° in this example, its subset portion corresponding to perspective image 250c is not available, but if the 360° panoramic image is instead acquired from acquisition location 265 (as discussed further below with respect to image angle descriptor 270), such a 360° panoramic image would include a subset portion having visual information similar to that of perspective image 250c.

[0038] The visual content of the panoramic image 255d can be analyzed to determine the location of the image acquisition location 265 in the living room, and optionally further determine the orientation / direction information of the panoramic image. For this example, the panoramic image 255d is captured in the living room of the house and includes 180° horizontal coverage around a vertical axis (e.g., a semicircle showing approximately the northern quarter of the living room), and wherein the x and y axes of the visual content of the image are aligned with corresponding horizontal and vertical information in the room (e.g., a boundary between two walls, a boundary between a wall and a floor, the bottom and / or top of windows and doors, etc.). In this example, the image capture begins with a camera orientation in a west direction corresponding to a relative starting horizontal direction of 0° of the panoramic image 255d and continues in a semicircle, wherein the relative 90° horizontal direction of the panoramic image then corresponds to the north direction, and the relative 180° horizontal direction of the panoramic image corresponds to the east direction. If the full 360° panoramic image is instead captured from this same starting direction, it will include the same directional information as described above for the 180° panoramic image, and will further include a relative 270° horizontal direction of the 360° panoramic image corresponding to a south direction, and a relative horizontal 360° ending direction of the 360° panoramic image returning to a west direction.

[0039] Figure 2D Further illustrated is angle information 265 of the panoramic image 255d, which can be determined by the ILMM system and used to identify the locations of various types of wall elements of the living room shown in the panoramic image 255d. In this example, the wall elements include doors, windows, room-to-room wall openings, and wall-to-wall boundaries, with corresponding visual identifiers displayed in the legend 269, and the floor plan excerpt 260d similarly illustrates the locations of such types of wall elements (including door 261, window 262, room-to-room wall opening 263, and wall-to-wall boundary 264) of the living room—it will be appreciated that other types of elements in the room can be used for image analysis instead, and information elements about such elements can be displayed on the floor plan in a variety of ways. While in some embodiments the locations of wall elements in the panoramic image and / or the locations of those wall elements in the living room can be automatically determined based on analysis of one or more images taken in the living room, in other embodiments, at least some of this information can be determined in other ways instead, such as specified by one or more human operators of the ILMM system or an associated system.

[0040] Regarding the angular information 265 of the panoramic image 255d, it includes a subset of information 265a showing the determined angular positions of the boundary elements between walls in the panoramic image 255d, wherein the first boundary element between walls (corresponding to the northwest corner of the living room) is visible in the panoramic image 255d at approximately 35° (relative to the westward starting direction of the panoramic image 0°), and wherein the second boundary element between walls (corresponding to the northeast corner of the living room) is visible in the panoramic image 255d at approximately 160° (also relative to the starting direction of the panoramic image). Similarly, the subset of information 265c shows the determined angular positions of the windows on the west and north walls of the living room in the panoramic image 255d, wherein a portion of the west window is visible from 0° to approximately 20° of the panoramic image 255d, and the north window is visible from approximately 130° to slightly less than 150° (both relative to the starting direction of the panoramic image). Thus, while some wall elements (such as the boundaries between walls) may be represented at a single angle, other wall elements (such as windows, doors, openings, etc.) may instead be represented across a range of angular positions. As shown in the information subsets 265b and 265d corresponding to the wall opening and door, respectively, the living room door and the living room wall opening are not visible in the panoramic image 255d, and therefore the corresponding angular position information for those types of wall elements is not identified in this example. In this example, the various angular position information subsets 265a-265d are combined to create an aggregate image angle descriptor 265 for the panoramic image 255d, which for illustrative purposes uses the same visual identifier for different types of wall elements, although the information may be encoded and stored in other formats (e.g., using text labels, numeric identifiers associated with corresponding wall element types, etc.) - additionally, such an image angle descriptor 265 may include information for some or all angles of the image (e.g., in this example, the vector has 181 values ​​corresponding to a separate value for each of the angles 0° to 180°).

[0041] exist Figure 2D, which is similar to descriptor 265 but corresponds to the case where the analyzed panoramic image instead has a 360° coverage starting at acquisition location 265, wherein the first half of descriptor 270 (i.e., the first 180° of angular information) is the same as descriptor 265, and wherein descriptor 270 further includes additional 180° angular information corresponding to the remainder of the living room. Thus, descriptor 270 includes further information about the determined angular positions of door 261 and wall opening 263, as well as the windows on the south wall and the inter-wall boundaries at the southeast and southwest corners of the living room - in addition, since the entire west window is now visible in the horizontal 360° coverage, additional portions of the west window are now represented in descriptor 270 with corresponding position information from approximately 350° to 360°.

[0042] As previously described, in various embodiments, information about the determined positions of the identified elements in the angle descriptors may be encoded and stored in various ways, including in the form of a vector having one or more values ​​for each directional angle, such as identifying each wall element present in a given angular direction - in other embodiments, other angular information other than a single horizontal degree may be represented in the image angle descriptor, such as less than a single degree or instead multiple degrees, and / or representing verticality (whether in lieu of or in addition to horizontality). Although not shown in these examples, it will be appreciated that in some cases, multiple wall elements may be visible from a particular room location at the same angular direction - if so, then in various embodiments, the angular position information in the corresponding angle descriptor for that single angular direction may be represented in various ways, such as including an indication of each type of visible wall element or instead indicating only or some subset of the visible wall elements. It will also be appreciated that many types of wall elements (such as doors and windows) will extend across multiple degrees of horizontal coverage, such as having windows in a northerly direction visible and identified in the resulting image angle descriptors for several angles. While in the above examples the panoramic image is captured with a starting direction facing west, it will be appreciated that in other situations the panoramic image may be captured in other ways—for example, other panoramic images may have a different starting direction, or if the panoramic image is instead captured with its entire horizontal coverage simultaneously (e.g., via one or more fisheye lenses), then a particular direction may be selected to be considered relative to 0° for the panoramic image (e.g., arbitrarily; by using a predefined direction, such as north; etc.).

[0043] In various embodiments, the angular position information of the wall elements in the panoramic image (or other types of images, such as perspective images) can be further determined in various ways. For example, in at least some embodiments, the ILMM system can perform automated operations to analyze the visual content of the image to identify one or more defined types of wall elements, including in some embodiments using machine learning techniques to identify specific types of wall elements in the visual content information of the image. Similarly, such automated analysis techniques can be used to determine the range of angular information covered by the visual content information of the image (e.g., a 180° range in the panoramic image 255d, or a 360° range in the panoramic image associated with the image angle descriptor 270), or instead in other embodiments such information can be determined in other ways (e.g., based at least in part on metadata associated with the image capture). Once the range of angular information is determined for the panoramic image, automated operations can be further used to determine specific angles associated with specific positions in the image corresponding to some or all of the specific wall elements. In other embodiments, some or all of this position information for wall elements and / or angular range information for the images may be determined in other manners, such as if confirmed by a human operator user of the ILMM system (e.g., via the ILMM system GUI, not shown) after initial automated position determination by the ILMM system or instead entirely specified by such a user.

[0044] Figure 2E continue FIG. 2A to FIG. 2D , and for reference purposes includes examples from Figure 2D A copy of the image corner descriptor 270 is provided. In addition, Figure 2E Further shown is a variety of additional information related to generating building corner descriptors for various room locations in the living room - while additional building corner descriptors may be similarly generated and used for some or all of the other rooms of the house (and in some cases for exterior areas surrounding or otherwise near the house), which are not shown in this example for the sake of brevity. An excerpt 260e of a floor plan of a living room is shown (with the same Figure 2Dd of the drawings), where the living room floor plan excerpt 260e includes a grid 268 of various room locations within the living room (e.g., a portion of a larger grid of building locations extending throughout the house, not shown), and further information 265 is shown to indicate the 360° panoramic image acquisition location corresponding to the image angle descriptor 270—it will be appreciated that the location within such a grid may be determined in a variety of ways (e.g., based on a defined number of rows and / or columns, based on a defined total number of room locations, based on defined horizontal and / or vertical distances between adjacent room locations, etc.), and in other embodiments, the set of room locations may have a form other than a grid (including, in some cases, being randomly or otherwise selected in an irregular manner). In at least some embodiments, a building angle descriptor will be generated for each of the room locations, such as for later use in comparing those building angle descriptors with the image angle descriptors for the image in order to determine which of the building angle descriptors best matches the image angle descriptor.

[0045] In this example, in Figure 2E 268—for example, grid location 268c5 is located near location 265 and has a corresponding building angle descriptor 278c5 shown. Other example room descriptors shown include a building angle descriptor 278j5 corresponding to grid location 268j5 (i.e., in the same column as 268c5, but in a different row), and a building angle descriptor 278a2 corresponding to grid location 268a2. In this example, each of the building angle descriptors uses a North direction to correspond to 0°, continuing in a clockwise manner through 360°—however, for illustrative purposes, the building angle descriptors 278 are shown in FIG. Figure 2E 270) to facilitate the reader's visual comparison of the similarities and dissimilarities of the building angle descriptors to the image angle descriptors 270. As is visually apparent, room descriptor 278c5 (room location 268c5 from a room location close to location 265) is more similar to image angle descriptor 270 than the other example building angle descriptors shown, as would be expected since small changes in room location will generally result in small differences in the angular information regarding the wall element locations.

[0046] In various embodiments, a building angle descriptor for a given room location may be generated in a variety of ways, including by using information provided with the corresponding generated floor plan about the location of the relevant wall element on the wall of the floor plan. Given this information, geometric techniques may be used to determine the angular amount from a given room location and a starting direction to a given location on the wall, such as a location corresponding to the horizontal start or end point of a door, window, or wall opening, or the location of a boundary between walls. For purposes of illustration, the north window in the living room excerpt 260e is labeled 269a, and the south window is labeled 269b, with the corresponding window elements in the image angle descriptor 270 and the building angle descriptor 278 being labeled in the same manner to facilitate visual comparison by the reader. In other embodiments, the location of the relevant wall element on the wall of the floor plan may be determined in other ways (e.g., if not provided with the generated floor plan), such as based on input from one or more human operator users of the ILMM system, from engineering drawings or other schematics of the building, and the like.

[0047] Figure 2F continue FIG. 2A to FIG. 2E , and for reference purposes includes examples from Figure 2D and Figure 2E A copy of the image corner descriptor 270 is provided. In addition, Figure 2F With Figure 2E The excerpt 260e further shows the floor plan of the living room in a similar manner, including an excerpt 260f showing the floor plan of the living room from Figure 2E The room location grid, but Figure 2FThe grid 288 in has additional information (e.g., similar to a heat map) about the degree of match of the associated building corner descriptor to the image corner descriptor 270 for each room location. In this example, the similarity / dissimilarity information 288 indicates that the grid room location 268c5 has the highest degree of match (e.g., highest similarity, lowest dissimilarity, lowest distance, etc.) to the image corner descriptor 270, while the room locations at row 3 and row 4 of column d and row 4 of column e have the next highest degree of match, and various other room locations generally decrease in their degree of match as their distance from room location 268c5 increases. In at least some embodiments, the comparison of the image corner descriptors to the building corner descriptors of the room can include: starting at a selected room location (e.g., randomly selected, selected at or near the center of the room, etc.), such as room location 268g4 in this example; and repeatedly moving in the direction of adjacent room locations with higher degrees of match using a nearest neighbor search until the best match is identified, as shown in excerpt 260f for room location 268c5. After determining the room location with the best match, the corresponding location in the room can be assigned as the determined acquisition location 289 of the 360° panoramic image, such as the room location of the best matching building corner descriptor in the example shown here, or in some embodiments, within a small distance from that room location (e.g., a distance calculated based on the amount and / or type of difference between the image corner descriptor and the best matching building corner descriptor). Once the acquisition location 289 of the 360° panoramic image is determined, it can be stored with the floor plan and / or otherwise used in one or more ways, as discussed in more detail elsewhere herein.

[0048] Although in FIG. 2D to FIG. 2F Not shown in the examples of , but in some embodiments and scenarios, acquisition location determination may be performed for images that may have been captured in any of a plurality of candidate rooms in one or more buildings - in various embodiments, such acquisition location determination activity may be performed in various ways, such as considering each possible room and finding the best matching room location across all of these rooms to narrow the set of possible candidate rooms before performing a match (e.g., by attempting to identify one or more elements that are visible in the image but are only present in one or a subset of the possible candidate rooms, etc.). In such embodiments, the grid of corner descriptors for a building may extend across some or all of the rooms of the building, and the corresponding search for the best match of the corner descriptors of the image to the corner descriptors of the rooms of the building may extend across the corner descriptors of multiple rooms (e.g., may include considering all corner descriptors generated for the building).

[0049] With respect to finding the best matching building angle descriptor for the image angle descriptor 270 from among multiple possible room locations in the room, in various embodiments, some or all of the building angle descriptors for the room locations in the grid may be compared to the image angle descriptors 270 in various ways to determine a match. For example, in some embodiments, a circular bulldozer distance metric may be used to compare two such descriptors in a rotation-independent manner, such that the two descriptors may have relative 0° starting directions pointing in different directions. In other embodiments, distance or other measures of similarity / dissimilarity may be used, such as by measuring the distance for each angle and aggregating that information across all angles.

[0050] Additionally, to facilitate comparison of two such angular descriptors in situations where the distance or similarity / dissimilarity metric is not rotation-independent, in some embodiments, additional automated operations may be performed to ensure that information encoded in a given relative angular direction in one angular descriptor is being compared with a relative angular direction in the other angular descriptor that points to the same actual real-world direction. For example, in some embodiments, a method may be used that compares each angular direction in one angular descriptor to a particular angular direction (e.g., a starting direction) in the other angular descriptor, thereby ensuring that one of the comparisons uses the same direction. Alternatively, in other embodiments, automated operations may be performed to synchronize the two angular descriptors to be compared, such as by identifying which relative angular directions in one angular descriptor correspond to which relative angular directions in the other angular descriptor (e.g., identifying for a relative 0° starting angular direction of one angular descriptor what the corresponding angular direction in the other angular descriptor is). With respect to Figure 2EFor example, such a determination may identify that the 0° start direction (corresponding to the west direction) of the image angle descriptor 270 is the same as the 270° direction (or -90° direction) in each building angle descriptor 278. Alternatively, in some embodiments, a limited number of instances of a feature in the environment may be identified, wherein the angular direction of each such instance of an angle descriptor is compared to the corresponding instance in another angle descriptor—an example of such a feature may be a direction orthogonal to or perpendicular to the plane of a wall (e.g., identified by vanishing point analysis using lines in the image), such as having 4 such instances for a 360° panoramic image in a typical rectangular room (i.e., one for each wall, which are approximately 90° apart from each other). As another alternative, such a feature in the environment may be a wall element type that has only one or a limited number of instances, and the angular direction in one angle descriptor for each such instance may be compared to the angular direction of an instance of the same wall element type in another angle descriptor—examples of such a feature may include doors (e.g., door start or end edges), inter-wall boundaries (e.g., typically having 4 such instances in a rectangular room), etc. In other embodiments, other distance metrics and / or similarity / dissimilarity metrics may be used, and other techniques may be used to synchronize corresponding information in two or more corner descriptors being compared.

[0051] Figure 2G continue FIG. 2A to FIG. 2F , and shows an example 255g of a 2D floor plan of house 198, such as may be presented to an end user in a GUI, where the living room is the most westerly room of the house (as reflected by direction indicator 209) - it will be appreciated that in some embodiments, a 3D or 2.5D floor plan showing wall height information may be similarly generated and displayed, either in addition to or in lieu of such a 2D floor plan. In this example, information 289 is added to floor plan 255g to indicate information about the living room. FIG. 2D to FIG. 2F The location of the determined acquisition location of the discussed 360° panoramic image includes orientation / direction information showing the panoramic image (e.g., to show that the panorama covers 360° and has a start / end direction in a westward direction). In other embodiments and scenarios, the location information of the image may be displayed in other ways, such as an example perspective image for the southwest side of the living room, which perspective image includes a visual indicator of the direction covered in the perspective image, and / or an additional panoramic image for the northeast corner of the living room, which shows the acquisition location in the room without showing the orientation / direction information of the panoramic image. When displayed as part of a GUI, the additional information 289 for the 360° panoramic image on the displayed floor plan can be a user-selectable control (or be associated with such a control) that allows the end user to select and display some or all of the associated 360° panoramic images (e.g., to be associated with FIG. 2A to FIG. 2D in a similar way).

[0052] In this example, various other types of information are also shown on the 2D floor plan 255g. For example, such other types of information may include one or more of the following: room labels added to some or all rooms (e.g., "living room" for the living room); room dimensions added for some or all rooms; visual indications of light fixtures or appliances or other built-in features added for some or all rooms; visual indications of the location of additional types of associated and linked information added for some or all rooms (e.g., other panoramic images and / or perspective images that the end user may select for further display, audio annotations and / or recordings that the end user may select for further presentation, etc.); visual indications of doors and windows added for some or all rooms; etc. In addition, in this example, a user-selectable control 228 is added to indicate the current floor displayed for the floor plan and allow the end user to select a different floor for display - in some embodiments, changes in floor or other levels may also be made directly from the floor plan, such as by selecting a corresponding connecting passage in the shown floor plan (e.g., stairs leading to floor 2). It will be appreciated that in some embodiments various other types of information may be added, that in some embodiments some of the types of information shown may not be provided, and that in other embodiments visual indications of and user selections of links and associated information may be displayed and selected in other manners.

[0053] Additional details regarding implementations of systems that provide at least some of the functionality of a MIGM system or related systems for generating floor plans and associated information and / or presenting floor plans and associated information are included in the following: co-pending U.S. non-provisional patent application Ser. No. 16 / 190,162 (Agent Docket No. 990074.403), filed Nov. 14, 2018, and entitled “Automated Mapping Information Generation From Inter-Connected Images” (which includes disclosure of an example floor map generation manager (or FMGM) system generally relating to automated operations for generating and displaying floor maps or other floor plans of a building using images captured in and around the building); and co-pending U.S. non-provisional patent application Ser. No. 16 / 190,162 (Agent Docket No. 990074.403), filed Nov. 14, 2018, and entitled “Automated Mapping Information Generation From Inter-Connected Images” (which includes disclosure of an example floor map generation manager (or FMGM) system generally relating to automated operations for generating and displaying floor maps or other floor plans of a building using images captured in and around the building); and co-pending U.S. non-provisional patent application Ser. No. 16 / 190,162 (Agent Docket No. 990074.403), filed Nov. 14, 2018, and entitled “Presenting Integrated Building Information Using Three-Dimensional Building Models” ( U.S. Non-Provisional Patent Application No. 16 / 681,787 (Attorney Docket No. 990074.407), entitled “Models” (including disclosures generally relating to example FMGM systems for automated operations for displaying floor maps or other floor plans of buildings and associated information); U.S. Non-Provisional Patent Application No. 16 / 841,581 (Attorney Docket No. 990074.408), entitled “Providing Simulated Lighting Information For Three-Dimensional Building Models” filed on April 6, 2020 (including disclosures generally relating to example FMGM systems for automated operations for displaying floor maps or other floor plans of buildings and associated information); U.S. Provisional Patent Application No. 62 / 927,032 (Attorney Docket No. 990074.409), entitled “Generating Floor Maps For Buildings From Automated Analysis Of Video Of The Buildings' Interiors” filed on October 28, 2019409P1) (which includes disclosures generally relating to example video-to-floor-map (or VTFM) systems for automated operations for generating floor maps or other floor plans of buildings using video data collected in and around the building); and U.S. Non-Provisional Patent Application No. 16 / 807,135 (Attorney Docket No. 990074.406), entitled “Automated Tools For Generating Mapping Information For Buildings,” filed on March 2, 2020 (which includes disclosures generally relating to example MIGM systems for automated operations for generating floor maps or other floor plans of buildings using images collected in and around the building); each of the above patent applications is incorporated herein by reference in its entirety. .

[0054] About Figures 2A to 2G Various details are provided, but it will be understood that the details provided are non-exclusive examples included for illustrative purposes and that other implementations may be performed in other ways without some or all of such details.

[0055] Figure 3 300 is a block diagram illustrating an embodiment of one or more server computing systems 380 executing an embodiment of an ILMM system 380, and one or more server computing systems 300 executing an embodiment of an ICA system 340 and a MIGM system 345—the server computing systems and ILMM systems may be implemented using a plurality of hardware components that form electronic circuits suitable for and configured to perform at least some of the techniques described herein when operating in conjunction. In the illustrated embodiment, each server computing system 300 includes one or more hardware central processing units ("CPUs") or other hardware processors 305, various input / output ("I / O") components 310, storage devices 320, and memory 330, wherein the illustrated I / O components include a display 311, a network connection 312, a computer-readable medium drive 313, and other I / O devices 315 (e.g., a keyboard, a mouse or other pointing device, a microphone, a speaker, a GPS receiver, etc.). Each server computing system 380 may include hardware components similar to those of server computing system 340 , including one or more hardware CPU processors 382 , various I / O components 382 , storage devices 385 , and memory 387 , although some details of server 340 are omitted in server 380 for the sake of brevity.

[0056] The server computing system 380 and the executed ILMM system 340 can communicate with other computing systems and devices via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.), such as: user client computing devices 390 (e.g., for viewing floor plans, associated images and / or other related information); ICA and MIGM server computing systems 300; one or more image acquisition mobile devices 360; optionally other navigable devices 395 that receive and use floor plans and determined image acquisition locations and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices); and optionally other computing systems not shown (e.g., for storing and providing additional information related to buildings; for capturing building interior data; for storing and providing information to client computing devices, such as additional supplemental information associated with the image and the building or other surrounding environment it covers; etc.).

[0057] In the illustrated embodiment, an embodiment of the ILMM system 389 is executed in memory 387 to perform at least some of the described techniques, such as using processor 381 to execute software instructions of system 389 in a manner that configures processor 381 and computing system 380 to perform automated operations that implement those described techniques. The illustrated embodiment of the ILMM system may include one or more components (not shown) to each perform a portion of the functionality of the ILMM system, and the memory may further optionally execute one or more other programs 391—as a specific example, in at least some embodiments, a copy of the ICA and / or MIGM system may be executed as one of the other programs 391, such as in place of or in addition to the ICA system 340 and the MIGM system 345 on the server computing system 300. The ILMM system 389 may further store and / or retrieve various types of data on the storage device 385 (e.g., in one or more databases or other data structures) during its operation, such as: various types of floor plan information and other building mapping information 386 (e.g., generated and saved 2D floor plans and the locations of wall elements and other elements on those floor plans, generated and saved 2.5D and / or 3D models, building and room dimensions for use with associated floor plans, additional image and / or annotation information, etc.); information 393 regarding additional images (whose acquisition locations will be determined) and associated information 392 regarding such determined acquisition locations; information 387 regarding generated building corner descriptors and image corner descriptors; and optionally various other types of information (e.g., linked panoramic information). The ICA system 340 and / or the MIGM system 345 may similarly store and / or retrieve various types of data on the storage device 320 (e.g., in one or more databases or other data structures) during their operation, and provide some or all of this information to the ILMM system 389 for its use (whether in a push and / or pull manner), such as various types of floor plan information and other building mapping information 326 (e.g., similar or identical to information 386), various types of user information 322, captured 360° panoramic image information 324 (e.g., for analysis to generate floor plans; provided to users of client computing devices 390 for display; etc.), and / or various types of optional additional information 328 (e.g., various analytical information captured by the ICA system relating to the presentation or other uses of one or more building interiors or other environments).

[0058] Some or all of the user client computing devices 390 (e.g., mobile devices), image acquisition mobile devices 360, other navigable devices 395, and other computing systems may similarly include some or all of the same types of components shown for server computing systems 300 and 380. As a non-limiting example, the image acquisition mobile devices 360 are each shown to include one or more hardware CPUs 361, I / O components 362, storage devices 365, imaging systems 364, IMU hardware sensors 369, and memory 367, wherein one or both of a browser 368 and one or more client applications 369 (e.g., applications specific to the ILMM system and / or ICA system) are executed within the memory 367, such as to participate in communications with the ILMM system 389, the ICA system 340, and / or other computing systems. Although specific components are not shown for the navigable device 395 or the client computing system 390, it will be appreciated that they may include similar and / or additional components.

[0059] It will also be appreciated that computing systems 300 and 380 and Figure 3 Other systems and devices included in the present invention are merely illustrative and are not intended to limit the scope of the present invention. The system and / or device may be changed to include multiple interactive computing systems or devices, and may be connected to other devices not specifically shown, including connecting via Bluetooth communication or other direct communication, through one or more networks (such as, the Internet), via the Web or via one or more dedicated networks (e.g., mobile communication networks, etc.). More generally, the device or other computing system may include any combination of hardware that can interact and perform the functions of the described types when programmed or otherwise configured with specific software instructions and / or data structures, the hardware including but not limited to desktop computers or other computers (e.g., tablet computers, tablet computers, etc.), database servers, network storage devices and other network devices, smart phones and other cellular phones, consumer electronic devices, wearable devices, digital music player devices, handheld game devices, PDAs, wireless phones, Internet appliances, and various other consumer products including appropriate communication capabilities. In addition, in some embodiments, the functions provided by the ILMM system 389 shown may be distributed in various components, some of the functions described in the ILMM system 389 may not be provided, and / or other additional functions may be provided.

[0060] It will also be appreciated that, although various entries are shown as being stored in memory or on a storage device when in use, for the purpose of memory management and data integrity, these entries or portions thereof may be transferred between memory and other storage devices. Alternatively, in other embodiments, some or all of the software components and / or systems may be executed in a memory on another device and communicate with the illustrated computing system via inter-computer communication. Therefore, in some embodiments, when configured by one or more software programs (e.g., an ILMM system 389 executed on a server computing system 380) and / or data structures, some or all of the described techniques may be performed by a hardware device including one or more processors and / or memories and / or storage devices, such as by executing software instructions of the one or more software programs and / or by storing such software instructions and / or data structures, and such as executing algorithms as described in flow charts and other disclosures herein. Furthermore, in some embodiments, some or all of the systems and / or components may be implemented or provided in other ways, such as by being comprised of one or more devices that are partially or completely implemented in firmware and / or hardware (e.g., rather than as devices that are implemented in whole or in part by software instructions that configure a particular CPU or other processor), including, but not limited to, one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), network storage device, or portable media article (e.g., DVD disk, CD disk, optical disk, flash memory device, etc.) for reading by an appropriate drive or via an appropriate connection. In some embodiments, the systems, components, and data structures may also be transmitted via generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on a variety of computer-readable transmission media, including wireless-based and wire / cable-based media, and may take a variety of forms (e.g., as part of a single or multi-channel analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Therefore, other computer system configurations may be utilized to practice embodiments of the present disclosure.

[0061] Figure 4 An example flow chart of an implementation of an ICA system routine 400 is shown. The routine may be implemented by, for example, Figure 1AICA system 160, Figure 3 The ICA system 340 and / or an ICA system as otherwise described herein may be performed, such as to capture 360° panoramic images and / or other images at a capture location within a building or other structure, such as for subsequent generation of related floor plans and / or other mapping information. Although portions of the example routine 400 are discussed with respect to capturing particular types of images at particular capture locations, it will be appreciated that this or similar routines may be used to capture video or other data (e.g., audio), either in lieu of or in addition to such images. In addition, although the illustrated embodiments capture and use information from the interior of a target building, it will be appreciated that other embodiments may perform similar techniques for other types of data, including for non-building structures and / or for information on the exterior of one or more target buildings of interest. In addition, some or all of the routines may be performed on a mobile device used by a user to capture image information, and / or performed by a system remote from such a mobile device.

[0062] The illustrated implementation of the routine begins at box 405, where an instruction or information is received. At box 410, the routine determines whether the received instruction or information indicates to collect data representing the interior of the building, and if not, continues to box 490. Otherwise, the routine proceeds to box 412 to receive an instruction (e.g., from a user of the image acquisition mobile device) to start an image acquisition process at a first acquisition location. After box 412, the routine proceeds to box 415 to perform an acquisition location image acquisition activity, so as to collect a 360° panoramic image for the acquisition location in the interior of the target building of interest, such as providing a horizontal coverage of at least 360° around a vertical axis. The routine may also optionally obtain annotations and / or other information about the acquisition location and / or surroundings from the user, such as for later use in presenting information about the acquisition location and / or surroundings.

[0063] After completing block 415, the routine continues to block 420 to determine whether there are more acquisition locations at which images are to be acquired, such as based on corresponding information provided by the user of the mobile device. If so, the routine continues to block 422 to optionally initiate the capture of link information (such as acceleration data) during the movement of the mobile device along a path of travel away from the current acquisition location and toward the next acquisition location in the interior of the building. As described elsewhere herein, the captured link information may include additional sensor data (e.g., from one or more IMUs or inertial measurement units on the mobile device or otherwise carried by the user) and / or additional video information recorded during such movement. Initiating the capture of such link information may be performed in response to an explicit instruction from the user of the mobile device or based on one or more automated analyses of information recorded from the mobile device. In addition, in some embodiments, the routine may further optionally monitor the motion of the mobile device during movement to the next acquisition location and provide one or more guidance cues to the user regarding the motion of the mobile device, the quality of the sensor data and / or video information being captured, the associated lighting / environmental conditions, the desirability of capturing the next acquisition location, and any other suitable aspects of capturing link information. Similarly, the routine may optionally obtain annotations and / or other information about the travel path from the user, such as for later use in presenting information about the travel path or resulting inter-panoramic connecting links. In block 424, the routine determines that the mobile device has reached a next acquisition location (e.g., based on an indication from the user, based on the user's forward movement stopping for at least a predefined amount of time, etc.) to serve as a new current acquisition location, and returns to block 415 to perform acquisition location image acquisition activities for the new current acquisition location.

[0064] If instead it is determined in box 420 that there are no longer acquisition locations at which image information of the current building or other structure is to be acquired, the routine proceeds to box 425 to optionally analyze the acquisition location information of the building or other structure, such as to identify possible additional coverage (and / or other information) to be acquired within the interior of the building. For example, the ICA system may provide one or more notifications to the user regarding the information acquired during the capture of multiple acquisition locations and optionally corresponding link information, such as if it is determined that the quality of one or more segments of the recorded information is insufficient or undesirable or does not appear to provide complete building coverage. After box 425, the routine continues to box 435 to optionally pre-process the acquired 360° panoramic images before they are subsequently used to generate related mapping information. In box 477, the images and any associated generated or obtained information are stored for later use. FIG. 5A to FIG. 5B One example of a routine for generating a floor plan representation of a building interior from such generated panoramic information is shown.

[0065] If it is instead determined in box 410 that the instructions or other information described in box 405 are not to capture images and other data representing the interior of a building, the routine instead continues to box 490 to perform any other indicated operations (such as, any housekeeping tasks) as appropriate, configure parameters for use in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device capturing the interior of one or more buildings, an operator user of an ICA system, etc.), obtain and store other information about users of the system, respond to requests for generated and stored information, etc.

[0066] After frame 477 or 490, routine advances to frame 495 to determine whether to continue, such as until receiving explicit termination instruction, or only when receiving explicit continuation instruction instead. If it is determined to continue, routine returns to frame 405 to wait for additional instructions or information, and if not, then advances to step 499 and ends.

[0067] FIG. 5A to FIG. 5B An example implementation of a flow chart for a mapping information generation manager (MIGM) system routine 500 is shown. The routine may be executed, for example, Figure 1A MIGM system 160, Figure 3 The MIGM system 345 and / or the MIGM system described elsewhere herein may be performed, such as generating a floor plan and optionally other mapping information (e.g., a 3D computer model) of the area based at least in part on the images defining the area. FIG. 5A to FIG. 5B In the example of , the generated mapping information includes a 2D floor plan and a 3D computer model of a building (such as a house), but in other embodiments, other types of mapping information can be determined and generated for other types of buildings and used in other ways, as discussed elsewhere in this document.

[0068] The illustrated embodiment of the routine begins at block 505 where information or instructions are received. The routine continues to block 510 to determine whether the instructions received in block 505 indicate that mapping information for the indicated building is to be generated, and if so, the routine continues to execute blocks 515 to 588 to do so, otherwise continuing to block 590.

[0069] In block 515, the routine determines whether image information is already available for the building, or whether such information needs to be collected instead. If it is determined in block 515 that the information needs to be collected, the routine continues to block 520 to collect such information, optionally waiting for one or more users or devices to move throughout the building and collect panoramic or other images at multiple collection locations in multiple rooms of the building, and optionally further analyzing the images and / or metadata information collected thereabout to interconnect the images, as discussed elsewhere herein— Figure 4 An example implementation of an ICA system routine for performing such an image acquisition is provided. If instead it is determined at block 515 that it is not necessary to acquire an image, the routine instead continues to block 530 to obtain existing panoramic or other images from multiple acquisition locations in multiple rooms of the building, optionally along with the acquisition of interconnected information of the images and metadata information related to movement between acquisition locations, such as may have been supplied in block 505 along with corresponding instructions in some cases.

[0070] After box 520 or 530, the routine continues to box 535 to optionally obtain additional information about the building, such as from activities performed during the acquisition and optional analysis of the images, and / or from one or more external sources (e.g., an online database, information provided by one or more end users, etc.)—such additional information may include, for example, the exterior dimensions and / or shape of the building, additional images and / or annotation information acquired corresponding to specific locations within the building (optionally for locations different from the acquisition location of the acquired panoramic or other image), and the like.

[0071] After block 535, the routine continues to block 550 to determine, for each room inside the building having one or more acquisition locations and associated acquired images, the room shape of the room, and optionally the location of its acquisition location within the room, for the data in the image taken inside the room, such as in an automated manner. In block 555, the routine further uses the visual data in the images and / or the acquisition metadata for these images to determine, for each room in the building, any connecting pathways into or out of the room (e.g., in an automated manner). In block 560, the routine further uses the visual data in the images and / or the acquisition metadata for these images to determine, for each room in the building, any wall elements in the room and their locations (e.g., in an automated manner), such as for walls, boundaries between walls, etc. It will be appreciated that while blocks 550 to 560 are shown as separate operations in this example, in some embodiments, a single analysis of the images may be performed to acquire or determine multiple types of information, such as those types of information discussed with respect to blocks 550 to 560.

[0072] In block 565, the routine then determines the estimated positions of the room shapes to create an initial 2D floor plan, such as by connecting the inter-room corridors in their respective rooms, by optionally positioning the room shapes around the determined acquisition locations (e.g., if the acquisition locations are interconnected), and by optionally applying one or more constraints or optimizations. Such a floor plan may include, for example, relative position and shape information for the various rooms without providing any actual size information for the individual rooms or the building as a whole, and may further include multiple linked or associated sub-maps of the building (e.g., to reflect different floors, levels, sections, etc.). The routine further associates the positions of doors, wall openings, and other identified wall elements on the floor plan.

[0073] After box 565, the routine optionally performs one or more steps 575 to 580 to determine and associate additional information with the floor plan. In box 575, the routine optionally estimates the dimensions of some or all of the rooms, such as based on analysis of the image and / or its acquisition metadata or based on overall dimensional information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan—it will be appreciated that if sufficiently detailed dimensional information is available, an architectural drawing, engineering drawing, etc. can be generated from the floor plan. After box 575, the routine continues to box 580 to optionally associate further information with the floor plan (e.g., with a specific room or other location within the building), such as additional image and / or annotation information. In box 585, the routine further estimates the height of the walls in some or all of the rooms, such as based on analysis of the image and optionally the size of known objects in the image, and height information about the camera when the image was acquired, and further uses this information to generate a 3D computer model of the building, wherein the 3D model and the floor plan are associated with each other.

[0074] After box 585, the routine continues to box 588 to store the generated mapping information and optionally other generated information, and optionally further use the generated mapping information, such as providing the generated 2D floor plan and / or 3D computer model for display on one or more client devices, providing the generated information to one or more other devices for automating navigation of those devices and / or associated vehicles or other entities, etc.

[0075] If it is instead determined in box 510 that the information or instructions received in box 505 are not to generate mapping information for the indicated building, the routine instead continues to box 590 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and responding to requests for previously generated computer models and / or floor plans and / or other generated information (e.g., requests for such information to be used by the ILMM system, requests for such information to be displayed on one or more client devices, requests for such information to be provided to one or more other devices for automated navigation, etc.), obtaining and storing information about the building for use in later operations (e.g., information about the size, number or type of rooms, total floor area, other adjacent or nearby buildings, adjacent or nearby vegetation, exterior images, etc.), etc.

[0076] After frame 588 or 590, routine continues to frame 595 to determine whether to continue, such as until receiving explicit termination instruction, or only when receiving explicit continuation instruction instead. If determined to continue, routine returns to frame 505 to wait and receive additional instructions or information, otherwise, continue to frame 599 and end.

[0077] Figure 6 An example implementation of a flow chart for an image location mapping manager (ILMM) system routine 600 is shown. The routine may be executed, for example, Figure 1A ILMM system 140, Figure 3 ILMM system 389 and / or as described in relation to Figures 2D to 2G and performing the ILMM system described elsewhere herein, such as performing automated operations related to determining an acquisition location of an image based at least in part on an analysis of image content and subsequently using the determined acquisition location information in one or more automated ways. Figure 6 In the example of , the collection location is determined relative to a floor plan of a building (such as a house), but in other embodiments, other types of mapping information may be used for other types of structures or for non-structural locations, and the determined collection location information may be used in other ways than discussed with respect to routine 600, as discussed elsewhere in the document.

[0078] The illustrated embodiment of the routine begins at block 605 where information or instructions are received. The routine continues to block 610 to determine whether the instructions received in block 605 indicate to determine the acquisition location of additional images of the indicated room and / or building, and if so, the routine continues to execute blocks 615 to 688 to do so, otherwise continuing to block 690.

[0079] In block 615, the routine obtains information about the room and / or building indicated in block 605, such as by receiving that information in block 605 or by otherwise retrieving stored information—the obtained information may include a floor plan of the building (or an excerpt of the floor plan of the indicated room) together with information about the locations of wall elements in the room, such as doors, inter-room wall openings, windows, inter-room boundaries, etc. In block 620, the routine then obtains or generates building description information for the indicated building or room (e.g., for each indicated room or each room in the indicated building), the building description information including a building angle descriptor for each of a plurality of room locations in the building / room (e.g., at the room location in a specified grid), the building angle descriptor identifying which wall elements, if any, are present in an angular direction (e.g., at each of 360 horizontal degrees of the angular direction) from the room location associated with the building angle descriptor.

[0080] In block 625, the routine then obtains information about an additional image (whose acquisition location is to be determined), such as by receiving the image in block 605 or by otherwise retrieving a stored copy of the image, and then proceeds to obtain or generate information about the locations of wall elements visible in the image, such as by analyzing the visual information content of the image. The routine then proceeds to block 630 to obtain or generate image description information for the image, including an image angular descriptor that identifies which wall elements, if any, are present in the visual content of the image at an angular direction corresponding to those visual contents (e.g., at each of the 360 ​​horizontal degrees of angular directions, such as relative to an angular direction determined to be a starting direction for the image, if the image is a 360° panoramic image).

[0081] In box 640, the routine then compares the image corner descriptor to some or all of the building corner descriptors (e.g., for all rooms, for one or more rooms to which the image may correspond, etc.) to determine a best matching building corner descriptor, such as a building corner descriptor having a minimum dissimilarity distance with the image corner descriptor. In box 685, the routine then identifies a room location to be used as a determined acquisition location for the image based on the room location associated with the best matching building corner descriptor, such as using the associated room location as the determined acquisition location. In some embodiments and scenarios, the routine may further determine orientation and / or direction information based on the determined acquisition location corresponding to one or more portions of the image (e.g., a starting direction of the image and / or an ending direction of the image).

[0082] After box 685, the routine continues to box 688 to store the information determined and generated in boxes 615 to 685, and optionally display the determined image acquisition location information of the image in the enclosed room on the floor map (or floor map excerpt), although in other embodiments, the determined information may be used in other ways (e.g., for automated navigation of one or more devices).

[0083] If it is instead determined in block 610 that the information or instructions received in block 605 are not to determine the acquisition location of the additional image, the routine instead continues to block 690 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving a request for previously determined image acquisition location information and / or for associated additional images (e.g., a request for such information to be displayed on one or more client devices, a request for such information to be provided to one or more other devices for use in automated navigation, etc.), obtaining and storing information about the building for use in later operations (e.g., information about floor plans and the locations of associated wall elements of rooms in the floor plans, etc.), performing geometric positioning techniques to test the association of wall elements visible in the image with wall elements present in the room (whether to confirm the match of a building corner descriptor that has been determined to be the best match for the image corner descriptor, and / or as part of identifying such a best matching building corner descriptor), using machine learning techniques to learn the best encoding to allow the image to be matched to the room location, etc.

[0084] After frame 688 or 690, routine continues to frame 695 to determine whether to continue, such as until receiving explicit termination instruction, or only when receiving explicit continuation instruction instead. If determined to continue, routine returns to frame 605 to wait and receive additional instructions or information, otherwise, continues to frame 699 and ends.

[0085] Figure 7 An example implementation of a flow chart for a building map viewer system routine 700 is shown. The routine may be executed, for example, Figure 1A The map viewer client computing device 175 and its software system (not shown), Figure 3 The client computing device 390 and / or a mapping information viewer or presentation system as described elsewhere herein may be executed, such as: receiving and displaying mapping information (e.g., a 3D computer model, a 2.5D computer model, a 2D floor plan, etc.) of a defined area, the mapping information including a visual indication of one or more determined image acquisition locations; and optionally displaying additional information (e.g., images) associated with a particular location in the mapping information. Figure 7In the example shown, the mapping information presented is for the interior of a building (such as a house), but in other embodiments, other types of mapping information may be presented for other types of buildings or environments and used in other ways, as discussed elsewhere in this document.

[0086] The illustrated implementation of the routine begins at block 705, where an instruction or information is received. At block 710, the routine determines whether the received instruction or information indicates displaying or otherwise presenting information representing the interior of the building, and if not, continues to block 790. Otherwise, the routine proceeds to block 712 to retrieve an indication of a floor plan and / or other generated mapping information of the building (e.g., a 3D computer model) and optionally associated linked information of locations within and / or around the building, and selects an initial view of the retrieved information (e.g., a view of a floor plan, at least some of the 3D computer model, etc.). In block 715, the routine then displays or otherwise presents a current view of the retrieved information, and waits for a user selection in block 717. Following the user selection in block 717, if it is determined in block 720 that the user selection corresponds to the current location (e.g., to change the current view), the routine continues to block 722 to update the current view in accordance with the user selection, and then returns to block 715 to update the displayed or otherwise presented information accordingly. User selections and corresponding updates to the current view may include, for example, displaying or otherwise presenting a user-selected piece of associated linked information (e.g., a particular image associated with a displayed visual indication of a determined acquisition location), changing the manner in which the current view is displayed (e.g., zooming in or out; rotating information where appropriate; selecting a new portion of a floor plan and / or 3D computer model to be displayed or otherwise presented, such as where some or all of the new portion was not previously visible, or instead where the new portion is a subset of previously visible information; etc.).

[0087] If it is instead determined in box 710 that the instructions or other information received in box 705 will not present information representing the interior of a building, the routine instead continues to box 790 to perform any other indicated operations (such as, any housekeeping tasks) as appropriate, configure parameters for use in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device capturing the interior of one or more buildings, an operator user of an ILMM system, etc.), obtain and store other information about users of the system, respond to requests for generated and stored information, etc.

[0088] After block 790, or if it is determined in block 720 that the user selection does not correspond to the current location, the routine proceeds to block 795 to determine whether to continue, such as until an explicit termination indication is received, or instead only if an explicit continuation indication is received. If it is determined to continue (e.g., if the user makes a selection related to a new location to be presented in block 717), the routine returns to block 705 to wait for additional instructions or information (or if the user makes a selection related to a new location to be presented in block 717, then proceed to block 712), and if not, proceed to step 799 and end.

[0089] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It will be appreciated that each frame shown in the flowchart and / or block diagram and the combination of frames shown in the flowchart and / or block diagram can be implemented by computer-readable program instructions. It will be further appreciated that in some embodiments, the functions provided by the routines discussed above can be provided in an alternative manner, such as being divided into more routines or merged into fewer routines. Similarly, in some embodiments, the routines shown can provide more or less functions than the functions described, such as when other routines shown are changed to lack or include such functions accordingly, or when the amount of the functions provided is changed. In addition, although various operations can be shown as being performed in a particular manner (e.g., serial or parallel or synchronous or asynchronous) and / or in a particular order, in other embodiments, operations can be performed in other orders and other ways. Any data structure discussed above can also be structured in different ways, such as by dividing a single data structure into multiple data structures and / or by merging multiple data structures into a single data structure. Similarly, in some embodiments, illustrated data structures may store more or less information than described, such as when other illustrated data structures are correspondingly altered to lack or include such information, or when the amount or type of information stored is varied.

[0090] In light of the foregoing, it will be appreciated that, although specific embodiments have been described herein for illustrative purposes, various modifications may be made without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited except by the corresponding claims and the elements described in those claims. In addition, although certain aspects of the present invention may be presented in certain claim forms at certain times, the inventors contemplate various aspects of the present invention in any available claim form. For example, although only some aspects of the present invention may be described as embodied in a computer-readable storage medium at a particular time, other aspects may also be embodied in this manner.

Claims

1. A computer-implemented method comprising: selecting, by a computing device, a plurality of room locations in the building by specifying a grid of room locations covering at least some of the plurality of rooms in the building; obtaining, by the computing device, visual description information for the building, the visual description information comprising building angle descriptors for the plurality of room locations in the grid of room locations covering the at least some of the plurality of rooms of the building, wherein each building angle descriptor is associated with one of the room locations and has angle information about a wall element of a wall that is visible from a specified angle direction of the associated room location; generating, by the computing device, an image angle descriptor for a panoramic image captured in a room of the building and comprising visual information about at least some of the walls of the room, wherein the image angle descriptor comprises information identifying wall elements of the walls of the room, the wall elements being visible in a specified direction within the visual information of the panoramic image; comparing, by the computing device, the image angle descriptors with the building angle descriptors to determine one of the building angle descriptors that is in the room and has angle information that best matches information included in the image angle descriptors, wherein the comparison of the image angle descriptors with the building angle descriptors comprises: performing a nearest neighbor search on the room positions of the grid, including identifying the determined one building angle descriptor by repeatedly moving from the at least one current room position to the at least one adjacent neighbor room position if the similarity of the at least one adjacent neighbor room position in the grid to the image angle descriptor is greater than the similarity of the at least one current room position in the grid to the image angle descriptor; associating, by the computing device and based on the comparing, the panoramic image with a determined location in the room and a determined orientation, the determined location comprising a room location with which the determined one building corner descriptor is associated, and the determined orientation identifying at least one direction from that room location corresponding to a specified portion of visible information in the panoramic image; and A floor plan of the building showing the room is presented, by the computing device and for the building, the floor plan having a visual indication identifying at least the determined location of the panoramic image, to cause navigation of the building using the presented floor plan.

2. The computer-implemented method of claim 1 , wherein: Presenting the floor plan also includes: visually indicating the determined orientation, and wherein the method also includes: presenting, by the computing device and in response to a user selecting the visual indication on the presented floor plan, at least a portion of the panoramic image corresponding to the determined orientation.

3. The computer-implemented method of claim 1 , wherein: The visual information of the panoramic image includes a 360-degree horizontal visual coverage from a capture position of the panoramic image, wherein, for each of the 360 ​​horizontal degrees of visual coverage from the acquisition position, the image corner descriptor comprises an indication of any wall element of a wall of the room that is visible in a direction corresponding to the horizontal degree of visual coverage from the acquisition position, and wherein, for each of the 360 ​​levels from the room location associated with the building angle descriptor, each of the building angle descriptors includes an indication of any wall elements of the walls of the surrounding rooms that are visible in a direction corresponding to the visual coverage of the level from that room location.

4. The computer-implemented method of claim 3, wherein: The comparing of the image corner descriptor with the building corner descriptor further comprises: analyzing the visual information to identify, for a specified type of feature of a wall of the room, at least one of the features present within a 360-degree horizontal visual coverage from the acquisition location; For each of at least some of the building corner descriptors, the image corner descriptor is compared with the building corner descriptor by: identifying the presence of one or more of said characteristics in said 360 horizontal degrees from said room location associated with said building corner descriptor; and synchronizing the position of each of the identified at least one of the 360-degree horizontal visual coverage from the acquisition position to the position of each of the identified one or more 360-degree horizontal visual coverage from the room position to determine whether information of the coverage at other horizontal degrees in the image angle descriptor matches information of the coverage at other horizontal degrees in the building angle descriptor relative to the synchronized position; and selecting one of the at least some of the building angle descriptors as the determined one building angle descriptor based on the selected one building angle descriptor having the identified synchronization position, and determining the orientation in the room for the panoramic image using the identified synchronization position, wherein for the identified synchronization position, information about coverage at other levels in the building angle descriptor best matches information about coverage at other levels in the image angle descriptor, and Wherein the characteristic of the specified type is one of: a visible wall orthogonal to a line along the identified horizontal visual coverage, or a wall element of the specified type visible at the identified horizontal visual coverage.

5. The computer-implemented method of claim 3, further comprising at least one of: For each of the building corner descriptors, wall elements of walls of surrounding rooms are determined for a room position associated with the building corner descriptor, wherein The wall elements of the walls of the surrounding room include at least one door, at least one window and at least one inter-wall boundary; or determining the one building corner descriptor having angle information that best matches information included in the image corner descriptor by performing the generating and the comparing without using any depth information acquired from any depth sensor regarding a depth from the acquisition position to a wall of a room in which the panoramic image was captured; or The comparison of the image corner descriptors with the building corner descriptors is performed by using machine learning to identify the determined one of the building corner descriptors as being most similar to the image corner descriptor.

6. The computer-implemented method of claim 1 , wherein: The comparison of the image angle descriptors with the building angle descriptors includes: for each of at least some of the building angle descriptors, determining the likelihood that the image angle descriptor and the building angle descriptor match because they differ by less than a specified threshold; and selecting one of the at least some of the building angle descriptors that has the highest likelihood of matching the image angle descriptor as the determined one of the building angle descriptors.

7. The computer-implemented method of claim 1 , wherein: The comparison of the image corner descriptors with the building corner descriptors includes: for each of at least some of the building corner descriptors: using a circular bulldozer distance measurement of the distance between the image corner descriptor and the building corner descriptor, and selecting one of the at least some of the building corner descriptors having a minimum measured distance to the image corner descriptor as the determined one building corner descriptor.

8. The computer-implemented method of claim 1, wherein: The associating of the panoramic image with the determined position and the determined orientation further comprises the computing device performing the following operations: generating additional visual information for the determined one building angle descriptor, the additional visual information representing a view from a room location with which the determined one building angle descriptor is associated and including, for the determined one building angle descriptor, wall elements of a wall of the room visible in the specified angular direction; as well as The visual information of the panoramic image is compared for similarity with the additional visual information generated for the determined one building corner descriptor to confirm that the similarity exceeds a specified threshold.

9. A non-transitory computer-readable medium having stored thereon content, the content causing one or more computing devices to perform automated operations, the automated operations comprising at least: selecting, by the one or more computing devices, a plurality of building locations in the building by specifying a grid of building locations covering at least some of a plurality of rooms in the building; obtaining, by the one or more computing devices and for an image including visual information about at least some of the buildings, an angular descriptor for the image, the angular descriptor including information identifying elements within the visual information that are visible from the buildings at a specified direction; obtaining, by the one or more computing devices, additional angular descriptors, each associated with one of the plurality of building locations in the grid covering the at least some of the plurality of rooms of the building and comprising angular information about an element visible from a specified angular direction of the associated building location; comparing, by the one or more computing devices, the angular descriptors for the image with the additional angular descriptors to determine one of the additional angular descriptors having angular information that best matches information included in the angular descriptors for the image, wherein the comparison of the angular descriptors for the image with the additional angular descriptors comprises: performing a nearest neighbor search for the building locations of the grid, including identifying the determined one additional angular descriptor by repeatedly moving from the at least one current building location to the at least one adjacent neighbor building location if the similarity of the at least one adjacent neighbor building location in the grid to the angular descriptor for the image is greater than the similarity of the at least one current building location in the grid to the angular descriptor for the image; associating, by the one or more computing devices, the image with a determined location of the building, the determined location being based on the associated building location for the determined one additional angular descriptor; and Information about the determined location of the building in the image is provided by the one or more computing devices.

10. The non-transitory computer readable medium of claim 9, wherein: The image is a panoramic image having 360 horizontal degrees of visual information, wherein obtaining the angular descriptors for the image comprises generating the angular descriptors by the one or more computing devices via analysis of the image, and wherein providing information about a determined location of the image comprises presenting a floor plan of the building, the floor plan including a visual indication of the determined location of the image.

11. The non-transitory computer readable medium of claim 9, wherein: The elements visible within the visual information for the building in the specified direction are wall elements including multiple doors or windows or boundaries between walls, wherein, for an additional angular descriptor determined, the elements visible from the specified angular direction of the associated building location are the wall elements, and wherein providing information about the determined location of the image includes: presenting a floor plan of the building, the floor plan including a visual indication of the determined location of the image.

12. The non-transitory computer readable medium of claim 9, wherein: The visual information of the image has a coverage range of less than 360 horizontal degrees, wherein the determined additional angular descriptor is for a panoramic image taken at the determined location and having a coverage range of 360 horizontal degrees, and wherein the comparison of the angular descriptors for the image with the additional angular descriptors includes: matching the angular descriptors for the image with a subset of the determined additional angular descriptors for the panoramic image.

13. A system for performing an automated operation, comprising: one or more hardware processors of one or more computing devices; as well as One or more memories storing instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations, the automated operations comprising at least: selecting a plurality of room locations of the rooms in the building by specifying a grid of room locations covering at least some of the rooms; obtaining description information of the room, the description information comprising building angle descriptors for the plurality of room locations in the grid covering the at least some of the room locations in the room, wherein each building angle descriptor is associated with one of the plurality of room locations and has angle information about an element identifiable in the room from a specified angle direction of the associated room location; generating an additional angular descriptor for information recorded at a recording location in the room, wherein the additional angular descriptor includes information identifying an element identifiable from the recorded information in the room from a specified direction of the recording location; comparing the additional angle descriptors to the building angle descriptors to determine one of the building angle descriptors having angle information that best matches the information included in the additional angle descriptors, wherein the comparison of the additional angle descriptors to the building angle descriptors comprises: performing a nearest neighbor search on the room positions of the grid, including identifying the determined one of the building angle descriptors by repeatedly moving from the at least one current room position to the at least one adjacent neighbor room position if the similarity of at least one adjacent neighbor room position in the grid to the additional angle descriptor is greater than the similarity of at least one current room position in the grid to the additional angle descriptor; Based on the comparison, associating the recorded information with a location in the room, the location determined for the recorded location based on the room location associated with the determined one building corner descriptor; and Information about the determined position in the room is provided for the recorded information.

14. The system according to claim 13, wherein: The recorded information includes a panoramic image with visual information, wherein the elements identifiable in the room from the recorded information include a set of wall elements having a multiple of doors or windows or wall boundaries visible in the visual information, wherein, for a determined building angle descriptor, the elements identifiable in the room from the specified angular direction of the associated room location include the wall elements, and wherein providing information about the determined position in the room includes: presenting a floor plan of the building including the room, wherein the presented floor plan includes a visual indication of the determined position in the room.

Images