Automatic exchange and use of attribute information between various types of building images

Abstract

Techniques are described for performing automated operations using computing devices to generate mapping information defining areas via analysis of visual data of images, including generating augmented images by using attribute information exchanged between pairs of images of various types or otherwise grouped images, and for using the generated mapping information in other automated manners, including using the generated mapping information for automated navigation and / or displaying or otherwise presenting the generated mapping information. In some cases, the defined areas include interiors of multi-room buildings, and the generated information includes at least one or more augmented images and / or partial floor plans and / or other modeled representations of the buildings, in some cases performed without having measured depth information regarding distances from the image's capture location to walls or other objects in the surrounding building.

Description

Technical Field

[0001] The following disclosure generally relates to techniques for automatically generating mapping information for defining a region by analyzing visual data from images of the region, such as by using attribute information exchanged between multiple types of images, and for subsequently using the generated mapping information in one or more ways, such as automatically generating and using floor plans and / or other modeled representations of buildings using images from inside buildings. Background Technology

[0002] In various fields and situations (such as building analysis, property inventory, real estate acquisition and development, renovation and alteration services, general contracting, and others), it may be desirable to view information about the interior of a house, office, or other building without having to physically visit or enter the building. This includes determining actual as-built information about the building rather than design information obtained before construction. However, it can be difficult to effectively capture, represent, and use such building interior information, including displaying visual information captured inside the building to users at remote locations (e.g., enabling users to fully understand the interior layout and other details, including controlling the display in a user-selectable manner). Furthermore, while floor plans can provide some information about the layout and other details of the building's interior, using floor plans in this way has several drawbacks in certain situations, including the difficulty in constructing and maintaining floor plans, the difficulty in accurately scaling and filling in information about room interiors, and the difficulty in visualizing and otherwise using them. Summary of the Invention

[0003] To address the aforementioned technical problems in the prior art, this application provides a system comprising: one or more hardware processors of one or more computing systems; and one or more memories having stored instructions, which, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform an automatic operation, the automatic operation comprising at least: acquiring multiple images of various types taken at multiple acquisition locations associated with a building, wherein, for each of multiple rooms in the building, the multiple acquisition locations include at least one acquisition location in the room; determining multiple image pairs from the multiple images, wherein each image pair includes a first image of a first type of the multiple types and an image of a second type of the multiple types. Different second types of second images, wherein the first image and the second image in each image pair have overlapping visual coverage of at least one of the plurality of rooms; for each of the plurality of image pairs, generating an enhanced image by modifying the first image of the image pair to use data associated with the second image of the image pair, including selecting at least one type of attribute associated with the second image of the image pair, and adding data associated with the modified first image for the selected at least one type of attribute; and providing at least some generated mapping information of the building for display, wherein the at least some generated mapping information is at least partially based on at least one generated enhanced image.

[0004] This application also provides a computer-implemented method comprising: acquiring, via one or more computing systems, multiple images of various types taken at multiple acquisition locations associated with a building, wherein the multiple images include multiple stereoscopic photographs and multiple panoramic images, and wherein, for each of multiple rooms in the building, the multiple acquisition locations include at least one acquisition location in the room; determining, via the one or more computing systems, multiple image pairs from the multiple images, wherein each image pair includes one of the multiple stereoscopic photographs and one of the multiple panoramic images, the one of the multiple stereoscopic photographs and the one of the multiple panoramic images having overlapping visual coverage of at least one of the multiple rooms; generating an enhanced image via the one or more computing systems and for each of the multiple image pairs by modifying a first image of the image pair to use data associated with a second image of the image pair, including selecting at least one type of attribute associated with the second image of the image pair, and adding data of the selected at least one type of attribute to the modified first image; generating mapping information of the building, at least in part based on the visual data of the multiple images, via the one or more computing systems and using at least one generated enhanced image; and presenting at least some of the generated mapping information of the building via the one or more computing systems. This application also provides a non-transitory computer-readable medium having stored content that enables one or more computing systems to perform automatic operations, the automatic operations including at least: acquiring multiple images of various types taken at multiple acquisition locations associated with a building by the one or more computing systems, wherein the multiple images of various types include multiple stereoscopic photographs and multiple panoramic images, and wherein, for each of multiple rooms in the building, the multiple acquisition locations include at least one acquisition location in the room;

[0005] Through the one or more computing systems, and for each of a plurality of image pairs, each image pair including one of the plurality of stereoscopic photographs and one of the plurality of panoramic images, the one of the plurality of stereoscopic photographs and the one of the plurality of panoramic images having overlapping visual coverage of at least one of the plurality of rooms, an enhanced image is generated by exchanging attribute data between the one stereoscopic photograph and the one panoramic image of the image pair, by: generating an enhanced stereoscopic photograph by modifying the one stereoscopic photograph in the image pair to use data of a first type of attribute from the one panoramic image in the image pair; and generating an enhanced panoramic image by modifying the one panoramic image in the image pair to use data of a second type of attribute from the one stereoscopic photograph in the image pair, wherein the first type of attribute and the second type of attribute are different; and by the one or more computing systems, providing at least some generated mapping information of the building for display, wherein the at least some generated mapping information is at least partially based on at least some of the generated enhanced images. Attached Figure Description

[0006] Figures 1A-1B This is a diagram illustrating an exemplary building interior environment and computing system used in embodiments of this disclosure, including images that automatically analyze visual data of various types of images acquired at acquisition locations within the building to determine images in which attribute information is exchanged to generate enhanced images, and portions that further present or otherwise use the enhanced images as mapping information of the building.

[0007] Figures 2A-2J The diagram illustrates an automated process of acquiring various types of images at acquisition locations on a building, analyzing the visual data of the images to determine which images are exchanged for attribute information to generate enhanced images, and subsequently generating and using the enhanced images in one or more automated ways.

[0008] Figure 3 This is a block diagram illustrating a computing system suitable for implementing at least some of the techniques described in this disclosure.

[0009] Figures 4A-4B An exemplary embodiment of a system routine for an Image Attribute Exchange and Mapping Information Generation Manager (IAEMIGM) according to an embodiment of the present disclosure is shown.

[0010] Figure 5 An exemplary embodiment of an image attribute exchange routine according to an embodiment of the present disclosure is shown, including a flowchart.

[0011] Figure 6An exemplary embodiment of a flowchart for a building information viewer system routine according to an embodiment of the present disclosure is shown.

[0012] Figure 7 An exemplary embodiment of a flowchart for an automatic image capture (AIC) system routine according to an embodiment of the present disclosure is shown. Detailed Implementation

[0013] This disclosure describes techniques for performing automated operations using computing devices, the automated operations involving generating mapping information for a defined area using images of the area, including by using attribute information exchanged between multiple types of images, and for subsequently using the generated mapping information in one or more other automated ways. In at least some embodiments, the defined area includes the interior of a multi-room building (e.g., a house, office, etc.), and the generated information includes at least partial floor plans of the building and / or other modeled representations of the building's interior, such as automated analysis of multiple images of various types acquired from various acquisition locations within the building, and optionally other data acquired in association with the building. In at least some such embodiments, generation is also performed without having or using depth information regarding measurements of distances from the image acquisition locations to walls or other objects in surrounding buildings. In various embodiments, the generated floor plans and / or other generated mapping-related information can be further used in various ways, including for controlling navigation of mobile devices (e.g., autonomous vehicles), for displaying on one or more client devices in a corresponding GUI (Graphical User Interface) using one or more visualization types, etc. The following includes additional details regarding the automatic generation, visualization, and use of such mapping information, and in at least some embodiments, some or all of the techniques described herein can be performed via the automated operation of the Image Attribute Exchange and Mapping Information Generation Manager (“IAEMIGM”) system, as discussed further below.

[0014] In at least some implementations and scenarios, various types of images acquired at the acquisition location associated with a building may include stereoscopic photographic images (e.g., photographs in stereoscopic format and photographs with limited viewing angles, such as those acquired without the use of a wide-angle lens and with a viewing angle of less than or equal to 60° or 90°, or those acquired using a wide-angle lens and with a viewing angle of less than or equal to 135°) and panoramic images (e.g., images with a wider viewing angle, such as greater than or equal to 180° or 360°, such as those acquired by using one or more fisheye lenses and / or other lenses, and optionally including rotation about a vertical axis or other axis, and optionally in an isorectangular format or other non-stereoscopic format), and different types of such images may have different associated benefits in different situations. As a non-exclusive example, stereoscopic photographs acquired for a building may have one or more types of preferred information relative to other panoramic images acquired for a building, such as having better chromaticity (or “color”) characteristics (e.g., due to better lenses and / or lighting, due to being captured by a professional photographer, etc.), to have higher resolution, thereby including more detail in a particular area of ​​interest to reflect areas of the building of particular interest and / or utility, etc. As another non-exclusive example, panoramic images acquired for a building may have one or more other types of preferred information relative to other stereoscopic photographs acquired for the building, such as having a wider visual coverage of surrounding rooms or other surrounding areas (e.g., if the panoramic image has 360° visual coverage around the vertical axis, it shows all the walls of the surrounding rooms, as well as parts or all of the floors and / or ceilings), and therefore have better correlated structural information determined by analyzing the visual data of the panoramic image (e.g., details of structural shapes or other structural elements, such as those for walls, floors, ceilings, windows, doors, and other wall openings, including the position of such structural information relative to each other and / or relative to the acquisition location of the panoramic image).

[0015] In some implementations and scenarios, the multiple types of images acquired at the acquisition location associated with a building may include other types of images, whether supplementary or alternative, having both stereoscopic and panoramic images. Examples of non-exclusive supplementary types of images include one or more of the following: daytime and nighttime images (e.g., multiple images of this type for one or more specific areas associated with a building, such as capturing a visual representation of the same subject during day and night), and the images are stereoscopic and / or panoramic images; images with multiple other types of lighting, such as natural and artificial lighting, and / or natural lighting at different times of day and / or years (e.g., multiple images of this type for one or more specific areas associated with a building, such as capturing a visual representation of the same subject during two or more periods of this type of lighting), and the image is a stereoscopic image. Photographs and / or panoramic images; images acquired at different times (e.g., multiple images of this type for one or more specific areas associated with a building, such as capturing a visual representation of the same subject at each of multiple different times), such as before or after an event (e.g., remodeling, repair, accident or other destructive event, construction, etc.), and the images are stereoscopic photographs and / or panoramic images; images of different types that respectively include non-visual data (e.g., information invisible to humans, such as for infrared light, ultraviolet light, radio waves and other radiation, audio, etc.) or visual data (e.g., multiple images of this type for one or more specific areas associated with a building, such as using visual data and one or more types of non-visual data to capture a representation of the same subject), and the images are stereoscopic photographs and / or panoramic images; etc.

[0016] As described above, the techniques described may include exchanging one or more types of attribute information between multiple types of images. For example, given a pair of associated images of multiple types (e.g., stereoscopic photographs and panoramic images with overlapping visual overlays), non-exclusive examples of exchanging attribute information between the pair of images may include one or more of the following: generating an enhanced image (e.g., an enhanced panoramic image) by modifying the panoramic image of the pair to use one or more types of data of attributes from the stereoscopic photograph of the pair (such as chromaticity data without luminance data, chromaticity data and luminance data, luminance data without chromaticity data, light balance data, saturation data, sharpness data, style data, etc.); generating an enhanced image (e.g., an enhanced stereoscopic photograph) by modifying the stereoscopic photograph of the pair to use or associate data of one or more types of attributes from the panoramic image of the pair, such as one or more types of structural shapes or other structural elements, information on the relative positions of elements optionally associated with each other (e.g., associating structural information such that the enhanced image has structural elements not visible in the original stereoscopic photograph), etc. Furthermore, other types of attribute data can be exchanged between two or more other types of images in a similar manner, whether replacing the example attribute types described above or in addition to those, such as generating one or more enhanced images that include a combination of visual data from the two or more images and / or a combination of attribute data using those two or more images. Non-exclusive examples include combining information from daytime and nighttime images (e.g., to illustrate changes between different times or otherwise provide a comparison of different lighting in the same area) and / or combining information from other types of images with different lighting (e.g., to illustrate changes between different lighting or otherwise provide a comparison of different lighting in the same area) and / or combining information from images captured at different times (e.g., to illustrate changes between different times or otherwise provide a comparison between different times) and / or combining information from images with captured visual and non-visual data or with different types of data (e.g., for illustrating changes between different types of data or for providing a comparison between different types of data), etc.

[0017] Furthermore, in some implementations, data associated with one or more images of the first type can be further used to additionally generate one or more enhanced images of the second type, wherein non-exclusive examples include one or more of the following: for a pair of stereoscopic photographs and panoramic images, using visual data from one image in the image pair with higher resolution and one or more corresponding trained machine learning models to generate an enhanced image for the other image in the pair, the enhanced image having a higher apparent resolution than some or all of the visual data of the enhanced image; for a pair of stereoscopic photographs and panoramic images, combining the visual data of the stereoscopic photographs and panoramic images to generate a new enhanced "fused" image, the new enhanced... A strongly "fused" image has visual data from two images and / or additional visual data relative to either a stereoscopic photograph or a panoramic image; for a pair of images of different types, the noise distribution of one image in the pair is used to enhance one or more parts of the resulting enhanced image based on the other image; for a pair of images of different types, information from the other image is combined with one or more parts of the enhanced image (e.g., as a point of interest in the enhanced image, such as the location where the other image is acquired in the enhanced image), and optionally user-selectable when the enhanced image is displayed to the user, such as displaying the other image, or otherwise presenting it to the user upon user selection. The image is associated with one of the images in a pair of images to generate an enhanced image; as part of at least some partial or complete floor plans or other modeled representations of a building generated based at least in part on a pair of images of different types, one or more locations within the floor plan or other modeled representation are associated with one or two of the image pairs (e.g., as one or more points of interest in the floor plan or other modeled representation, such as at one or more locations in the floor plan or other modeled representation, acquiring one or two images at one or two locations, and optionally, when the floor plan or other modeled representation is displayed to the user, it is user-selectable to display the corresponding image, or when the user selects, to provide the user with additional details associated with the image); with respect to multiple sets of images of various types acquired at multiple acquisition locations of the building used to generate one or more initial enhanced images, the image sets are updated at a later time to include one or more additional images (e.g., at one or more additional acquisition locations, one or more additional types, etc.), and one or more additional enhanced images are generated using the additional images (e.g., based on one or more new image pairs, each image pair including a pre-existing image and an additional image with overlapping visual coverage; based on one or more new image pairs, each image pair including two additional images with overlapping visual coverage, etc.).The system may, and optionally, update other mapping information of the building based on additional images; analyze multiple images of the first type (e.g., images with smaller visual coverage, such as stereoscopic photographs) to train one or more machine learning models to identify visual features included in those stereoscopic photographs (e.g., identifying features of areas of the building with additional detail that are of interest for further automated processing and / or for viewers, such as the kitchen or bathroom of a house, and optionally be able to determine an assessment corresponding to the current house or related to remodeling based on those additional details; identifying features of areas of the building that otherwise provide useful information for further automated processing and / or for the observer, such as providing information related to traffic flow areas between rooms, or providing information about the structure or layout of the building; identifying features of visually pleasing areas of the building to the observer, such as providing one or more introductory or overview images of the building; etc.), and generate one or more augmented images from the second type of images (e.g., images with larger visual coverage, such as panoramic images) using the trained machine learning models by selecting a subset of each of one or more images of the second type as one or more augmented images and corresponding to portions of the second type of images having the identified visual characteristics; etc. Furthermore, in at least some implementations and situations, unlike having a pair of images with overlapping visual coverage, the described technique can be extended to having a group of three or more images with overlapping visual coverage (e.g., a panoramic image and two or more stereoscopic photographs, such as having visual data corresponding to different subsets of the panoramic image; one or more stereoscopic photographs and two or more panoramic images, such as having visual data of two or more panoramic images overlapping each other, and each stereoscopic photograph overlapping at least a portion of at least one of the panoramic images; etc.), exchanging corresponding attribute information between the images in the group in a manner similar to that discussed for a pair of images (e.g., combining the chromaticity attribute data of two or more stereoscopic photographs in the group, such as via weighted or unweighted averaging, and using the combined chromaticity attribute data as part of an enhanced panoramic image of the panoramic image in the group).

[0018] Furthermore, in at least some implementations and situations, the described techniques include matching two or more types of images to be associated in a specific manner. For example, for a pair of related images including a stereoscopic photograph and a panoramic image, the matching of these images may be based at least in part on identifying overlapping visual overlays between the two images. As an example implementation, if one or more panoramic images are available for a building, each such panoramic image in a non-stereoscopic and / or non-planar format (e.g., isorectangular, spherical, etc.) can be analyzed first to generate multiple sub-images, each sub-image comprising a different subset of the visual data of the panoramic image, and being in a stereoscopic and / or planar format (e.g., generating six sub-images corresponding to a 360° panoramic image along the X, Y, and Z axes). For each such panoramic sub-image (and optionally for each panoramic image already in a stereoscopic format), further automated processing can be performed to generate one or more first global features describing the visual data of the sub-image (or the stereoscopic panoramic image) as a whole, and to generate multiple first local features describing the various portions of the visual data of the sub-image (or the stereoscopic panoramic image). Various techniques can be used to generate such global and local features, with an exemplary technique discussed in Yafei Lv et al., “An End-to-End Local-Global-Fusion Feature Extraction Network for Remote Sensing Image SceneClassification,” *Remote Sensing*, 2019, 11(24): 3006. Automated processing can similarly include generating one or more second global features that describe the visual data of each such stereo image as a whole, and generating multiple second local features that describe the individual parts of the visual data of each such stereo image, if one or more stereo images are available for a building. For each such stereo image, a group of one or more candidate panoramic images can then be determined by comparing one or more second global features of the stereo image with first global features generated for panoramic images and their sub-images, in order to rank the candidate panoramic images according to their degree of matching with those panoramic images and / or their sub-images, and optionally selecting a subset of candidate panoramic images that satisfy a defined threshold (e.g., the top 1, 5, or 10 above a defined matching level, etc.). If multiple candidate panoramic images are selected, the automatic processing can further compare multiple second local features of the stereoscopic photograph with multiple first local features of the candidate panoramic images (and their sub-images), and select one or more candidate panoramic images that have the highest degree of matching between the first and second local features.The stereoscopic photograph and those selected panoramic images can then be matched in pairs or other groups for further analysis, including regarding the exchange of attribute data between the images in the pair or other groups. Furthermore, in at least some embodiments and scenarios, multiple stereoscopic photographs matched to the same panoramic image in different pairs can also be combined into larger groups. Moreover, while in at least some embodiments and scenarios two or more images in a pair or other group can be selected at least partially based on overlapping visual coverage, in other embodiments, two or more such images to be paired or otherwise grouped can be selected in other ways, such as based on proximity (e.g., having acquisition locations in the same room or other area) and / or acquisition time (e.g., being captured during the same session and / or on a similar day) and / or one or more other indicative criteria.

[0019] The following includes exchanging attribute data between two or more paired or otherwise grouped images of various types, and using corresponding enhanced images in various ways (including automated operation regarding the implementation of the IAEMIGM system, and regarding...). Figures 2A-2J The examples and their associated descriptions discuss some of these details (as well as additional details related to this).

[0020] Furthermore, in some embodiments, supplemental visual data of the building, such as one or more videos, may be captured and used, although in other embodiments such supplemental visual data may not be used. Additionally, in some embodiments, supplemental capture metadata regarding image capture may be obtained and used in various ways, such as data obtained from IMU (Inertial Measurement Unit) sensors or other sensors of the mobile device when images are captured at the capture location and / or when the mobile device is carried by a user or moves between capture locations, although in other embodiments such capture metadata may not be used. As a non-exclusive example, supplemental location-related data may be obtained and used as part of the described techniques, for example, to determine the image acquisition location and / or capture orientation (e.g., image pose), and this information may be used as part of matching image pairs. Such location-related data can be based, for example, on sensors that directly provide location data (e.g., GPS sensors) and / or on a combination of multiple types of information, such as by combining acquired visual data and IMU data, including by using visual-inertial odometry techniques in a manner similar to ARKit and / or ARCore, and / or using one or more of Simultaneous Localization and Mapping (SLAM), Visual SLAM (V-SLAM), Visual-Inertial SLAM (VI-SLAM), Structure from Motion (SfM), etc. As another non-exclusive example, supplementary depth-related data can be obtained and used as part of the described techniques to help determine the pose of images and to use this information as part of matching image pairs. This depth-related data can be based, for example, on data acquired by one or more types of sensors and / or acquired data, such as using one or more LiDARs, time-of-flight, passive stereo, active stereo, structured lighting, etc. In some implementations, various other types of supplementary information may also be collected and used, and additional details relating to the acquisition and use of images and, optionally, other relevant information about buildings are included below. In at least some implementations, some or all of the relevant activities are performed via automated operation of an automatic image capture (“AIC”) system, as discussed further below.

[0021] As described above, in various embodiments, various types of mapping information for a building can be generated, at least in part, based on visual data from multiple images (including multiple images of various types) acquired for the building. For example, after acquiring multiple images of the building's interior (and optionally the building's exterior) and optionally other supplementary information, generating mapping information for the building (e.g., at least partial floor plans, linked sets of images at determined relative locations, enhanced images, etc.) may include automatically determining the relative positions of some or all of the image acquisition locations relative to each other in a common local coordinate system or other common local reference system, and optionally attempting to predict or otherwise determine the relative global positions of all acquisition locations relative to each other in a common global coordinate system or other common global reference system. Once such relative positions are determined, the orientation and relative distance between some or all pairs of acquisition locations can be determined. Regarding such relative distances, this determination may, for example, include identifying third and fourth acquisition locations that are twice as far apart as the first and second acquisition locations, but without knowing the actual distance between these acquisition locations. Similarly, regarding such relative orientation, determination may include, for example, identifying a first acquisition position to the right of a second acquisition position in a 60° direction (e.g., using the estimated pose orientation of the image acquired at the second acquisition position, or both the acquisition position and the capture orientation as the starting direction), and identifying a third acquisition position to the left of the second acquisition position in a 45° direction, but without knowing the actual geographic location of any of those acquisition positions. In various implementations, the determination of the relative positions of some or all image acquisition positions can be performed in various ways, including analyzing visual data from the images to interconnect some image pairs and / or corresponding pairs of these image acquisition positions (e.g., by identifying common matching features in two different images to determine their relative positions to each other, such as identifying the location and orientation of those images captured in part based on estimated pose information), and optionally using other supplementary information if available (e.g., metadata from image acquisition; other visual data; other information from the building, such as an overview image of the building or other information about the building, such as shape and / or size; etc.).

[0022] Furthermore, if performed in a particular embodiment, generating at least a partial floor plan of the building may further include: for each room in the building, and using visual data including one or more images of some of the interior of that room, the partial or complete structural shape visible in those images of that room, to correspond to structural elements such as one or more walls, floors, ceilings, passageways between rooms (e.g., doorways and openings between other walls), windows, fireplaces, islands, countertops, etc., and optionally to at least some non-structural elements (e.g., appliances, furniture, etc.). Generation may also determine the relative spacing between the multiple structural shapes of the rooms, such as based at least in part on the determined relative positions of the acquisition locations of those images and the estimated relative distances and orientations of those structural shapes from those acquisition locations. In some cases, the determination of the relative spacing between the structural shapes of multiple rooms may be further performed, such as based at least in part on the location of any connecting passageways between the rooms and / or using one or more images, each image having visual data including portions of multiple rooms. Such connecting passageways between rooms may include one or more of doorways, windows, staircases, non-room corridors, etc., and the automatic analysis of the visual data of the images may be based at least in part on identifying the contours of the passageways, identifying content within the passageways that differs from the exterior of the passageways (e.g., different colors, shading, light intensity, height, etc.) to identify such features. In some embodiments, generating at least a partial floor plan of the building may also include applying one or more types of constraints, including those based on connecting passageways between rooms (e.g., to coordinate positioning in two or more rooms connected by the passageway or otherwise match connecting passageway information), and optional other types of constraints (e.g., locations where rooms outside the building should not be located, the shapes of adjacent rooms, the total dimensions of the building and / or specific rooms within the building, some or all of the building's external shape, etc.). In some embodiments, and where the building has multiple floors or multiple levels, the connecting passageway information may be further used to associate corresponding portions on different sub-floor plans of different floors or levels. It should be understood that if enough images are captured to generally have visual data of all interior structural surfaces of the building, the generated floor plan may be a complete floor plan. In other cases, a predicted complete floor plan may be generated by predicting missing portions from one or more partial floor plans. Furthermore, in at least some implementations, the automatic analysis of images can be further performed by using machine learning (e.g., via a correspondingly trained machine learning model) to identify some or all of such information and / or additional information (e.g., estimated room type), such as estimating room type by identifying features or characteristics corresponding to different room types and associating corresponding semantic labels with such rooms.In other embodiments, at least some of this information may be obtained in other ways, such as by receiving relevant information from one or more users (e.g., user annotations based on one or more images of a room and / or other descriptions of a particular room or other location, such as identifying boundaries between walls, ceilings, and floors; based on other user input, such as adjustments to automatically determined information; etc.). In some embodiments, automated analysis of the visual data of the images may further identify additional information in one or more images, such as the dimensions of objects (e.g., objects of known size) and / or some or all of the dimensions of a room, and the estimated actual distance from the image acquisition location to the walls or other features in the room. For example, estimated size information of one or more rooms may be associated with floor plans and / or enhanced images, stored, and optionally displayed. If the height information of one or more rooms is estimated, some or all of the 3D (three-dimensional) models of the 2D (two-dimensional) floor plan may be created, associated with the floor plan, stored, and optionally displayed, and if size information for all rooms is generated with sufficient accuracy, more detailed floor plans of the building may also be generated, such as those with sufficient detail to allow for the generation of blueprints or other architectural plans. In various implementations, various identified or otherwise acquired information may also be associated with enhanced images, floor plans, and / or other generated building mapping information, and such enhanced images, floor plans, and / or other generated building mapping information (optionally including associated information) may be displayed or otherwise presented or otherwise provided to users and, optionally, other recipients in various ways. The following includes mapping information identifying the building, and additional details regarding the presentation or other use of such identified mapping information.

[0023] The described techniques offer various benefits in various implementations, including allowing the automatic generation of enhanced images and / or floor plans and / or other modeled representations of multi-room buildings and other structures based on images acquired in buildings or other structures. This includes, in at least some such implementations, the absence or non-use of depth information measured from depth sensors or other distance measuring devices regarding distances from the image acquisition location to walls or other objects in surrounding buildings or other structures, and for the presentation and / or other use of such enhanced images and / or floor plans and / or other modeled representations. The described techniques can further provide more complete and accurate room shape information and a wider variety of environmental conditions (e.g., views of individual images where objects in the room obscure at least some of the walls and / or floors and / or ceilings, but where the combination of views from multiple images eliminates or reduces this problem, etc.). Non-exclusive examples of further benefits of this described technology include the following: the ability to analyze visual data of target images to detect objects of interest (e.g., structural wall elements such as windows, doorways, and other wall openings) in an enclosed room, and the ability to determine the location of those detected objects within a defined room shape of the enclosed room; the ability to analyze additional captured data (e.g., motion data from one or more IMU sensors, visual data from one or more image sensors, etc.) to determine the path of the image acquisition device in multiple rooms, to identify wall openings (e.g., doorways, stairs, etc.) based at least in part on the additional data (and optionally on the visual data of one or more target images acquired in one or more rooms), and optionally to further use this information about the identified wall openings to align the defined 3D room shapes of the multiple rooms; the ability to interconnect multiple target images and / or their acquisition locations and to display at least one target image with one or more visual indicators (e.g., user-selectable visual indicators that, when selected, result in the display of a corresponding other image and / or associated information of that other image); etc. Furthermore, this automated technology allows such floor plans and / or other modeled representations to be generated much faster than previous technologies, and in at least some embodiments, with greater accuracy, at least in part based on the use of information acquired from the actual building environment (rather than from floor plans about how the building should theoretically be constructed), and enables the capture of changes to structural elements that occur after the initial construction of the building (e.g., remodeling and other updates). This described technology also offers the benefit of allowing mobile devices (e.g., semi-autonomous or fully autonomous vehicles) to perform improved automated navigation of buildings, including a significant reduction in the computing power they use and the time spent attempting to learn the building layout.Furthermore, in some embodiments, the described techniques can be used to provide an improved GUI where users can obtain more accurate and faster information about the interior and surrounding environment of a building (e.g., for navigating the interior and / or surrounding environment), including in response to search requests, as part of providing users with personalized information, as part of providing users with value estimates and / or other information about the building, etc. Various other benefits are also provided by the described techniques, some of which are further described elsewhere in this document.

[0024] For illustrative purposes, some embodiments are described below, wherein specific types of structures are used, and specific types of information is acquired, used, and / or presented in a specific manner using specific types of devices. However, it will be understood that the described techniques can be used in other ways in other embodiments, and therefore the invention is not limited to the exemplary details provided. As a non-exclusive example, although specific types of images are acquired and used to generate specific types of data structures (e.g., enhanced images, graphics of interconnected images and / or image acquisition locations, 2D floor plans, 2.5D or 3D computer models, queues, caches, databases, etc.), and these data structures are further used in a specific manner in some embodiments, it should be understood that other types of information describing buildings and their acquisition locations can be similarly generated and used in other embodiments, including for buildings (or other structures or layouts) separate from the house, and images and other building information can be used in other ways in other embodiments. As another non-exclusive example, although enhanced images and / or floor plans of houses or other buildings can be displayed to help viewers navigate the building, the generated mapping information can be used in other ways in other embodiments. As another non-exclusive example, while some embodiments discuss acquiring and using data from one or more types of image acquisition devices (e.g., mobile computing devices and / or separate camera devices), in other embodiments, the one or more devices used may take other forms, such as mobile devices that use some or all of the acquisition of additional data but do not provide their own computing capabilities (e.g., additional "non-computing" mobile devices), multiple separate mobile devices, each of which acquires some of the additional data (whether mobile computing devices and / or non-computing mobile devices), etc. Furthermore, the term "building" herein refers to any partially or completely enclosed structure, typically but not necessarily including one or more rooms that visually or otherwise separate the interior spaces of the structure. Non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, apartments, office buildings, commercial buildings or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.), supplementary structures on property together with another main building (e.g., separate garages or sheds on property with houses), etc. As used herein, the terms “acquisition” or “capture” relating to the interior of a building, acquisition location or other location (unless the context clearly indicates otherwise) may refer to the recording, storage or input of any media, sensor data and / or other information relating to spatial and / or visual characteristics and / or other perceptible characteristics of the interior of a building or a subset thereof, such as by a recording device or by another device receiving information from a recording device.As used herein, the term "panoramic photograph" or "panoramic image" can refer to a visual representation based on, including, or divisible into multiple discrete component images derived from substantially similar physical locations in different directions, and depicting a wider field of view than any single discrete component image depicts, including images from physical locations with a sufficiently wide field of view to include angles beyond what a person can perceive from a single direction of gaze (e.g., greater than 120°, 150°, or 180°, etc.). As used herein, the term "sequence" of acquisition locations generally refers to two or more acquisition locations, each of which has been visited at least once in a corresponding order, regardless of whether other non-acquisition locations have been visited in between, and regardless of whether the access to the acquisition locations occurs during a single consecutive time period or at multiple different times, or by a single user and / or device or by multiple different users and / or devices. Furthermore, various details are provided in the figures and text for illustrative purposes, but these details are not intended to limit the scope of the invention. For example, the dimensions and relative positions of elements in the figures are not necessarily drawn to scale, and some details have been omitted and / or provided more prominently (e.g., via size and positioning) to enhance readability and / or clarity. Furthermore, the same reference numerals may be used in the accompanying drawings to identify the same or similar elements or actions.

[0025] Figure 1A These are example block diagrams of various computing devices and systems that can participate in the described techniques in some implementations. Specifically, after mobile-type images are captured, such as by one or more mobile image acquisition computing devices 185 and / or one or more camera devices 186, the images and their associated information (e.g., annotations, metadata, interconnection information, etc.) can be stored on one or more server computing devices 180 along with information in image information storage device 164 for later use. Such information in image information storage device 164 can also be included as part of captured building interior information 165, which is subsequently used by an IAEMIGM (Image Attribute Exchange and Mapping Information Generation Manager) system 160 (whether on the same or different server computing system where the information in image information storage device 164 is stored) running on one or more server computing devices 180 to generate corresponding enhanced images of the building (e.g., at least in part based on image attribute data 150 from the images) and optionally other building mapping information 155 (e.g., submaps of linked acquisition locations, floor plans, etc.). Figure 2J An example of such a floor plan is shown below, and additional details relating to the automated operation of the IAEMIGM system are included elsewhere in this document, including regarding Figures 4A-4B and Figure 5The captured building interior information 165 may also include other types of information collected from the building environment, such as additional visual data and / or other types of data captured inside or near the building, as discussed in more detail elsewhere in this document.

[0026] exist Figure 1A In the illustrated embodiment, the capture of some or all images can be performed using an AIC (Automatic Image Acquisition) system executed on the mobile image acquisition computing device 185, such as application 162 located on the device's memory and / or storage device 152. In other embodiments, all images may be captured without using such an AIC system, regardless of whether the mobile image acquisition computing device and / or other camera device 186 lacks some or all of such computing power. If a copy of the AIC system used on the mobile image acquisition computing device 185 assists in image capture, one or more hardware processors 132 of the mobile device can execute the AIC system to acquire various images 143 and optionally associated additional information using one or more imaging systems 135 of the mobile device, which are then transmitted via one or more computer networks 170 to image information storage device 164 on server computing device 180. Similarly, if camera device 186 is used as part of image capture, the acquired image can be stored in the camera device's memory (not shown) and transmitted via one or more computer networks 170 to image information storage device 164 on server computing device 180, such as if the camera device includes a corresponding transmission capability (not shown) or directly transmitted by the camera device after the image has been transmitted to another device (not shown) including such a transmission capability. As part of the operation of the AIC system, various other hardware components of the mobile image acquisition computing device 185 can be further used, such as display system 142 (e.g., displaying instructions and / or component image information), device I / O components 136 (e.g., receiving instructions from the user and presenting information to the user), sensor module 148 including IMU gyroscope 148a and IMU accelerometer 148b and IMU compass 148c (e.g., to acquire sensor data and associate it with the acquisition of a specific corresponding component image), one or more illumination components 136, etc. Similarly, in some embodiments and situations, camera device 186 may include some or all of such components. Figure 1B An example of the acquisition of such images for a specific building 198 is shown, and Figures 2A-2C An example of this type of image is shown.

[0027] One or more users (not shown) of one or more client computing devices 105 may optionally interact with the IAEMIGM system 160 via computer network 170 to assist in creating or modifying building mapping information and / or subsequently using the generated mapping information in one or more other automated ways. Such user interaction may include, for example, providing instructions for generating building mapping information, providing information to include the generated building mapping information, obtaining specific generated mapping information and / or additional associated information, and optionally interacting with specific generated mapping information and / or additional associated information. Furthermore, one or more users (not shown) of one or more client computing devices 175 may optionally interact with server computing device 180 via computer network 170 to retrieve and use generated enhanced images and / or other generated building mapping information and / or individual images and / or other information associated with such generated building mapping information. Such user interaction may include, for example, obtaining and optionally interacting with one or more types of visual interaction with the generated mapping information of one or more buildings, optionally as part of a GUI displayed on such client computing devices. Furthermore, the generated mapping information (or a portion thereof) may be linked to or otherwise associated with one or more other types of information, including floor plans or other generated mapping information for multi-story or other multi-story buildings, to have multiple associated ground floor plans or other associated building mapping information subgroups for interconnecting different floors or floors (e.g., by connecting stairwells), for two-dimensional ("2D") floor plans of the building to be linked to or otherwise associated with the three-dimensional ("3D") presentation of the building, for 2D floor plans of the building to be linked to and / or 3D models and / or one or more component images for generating enhanced images or otherwise associated with 2D floor plans and / or 3D models and / or one or more component images, etc. Furthermore, although in Figure 1A Although not shown in the diagram, in some embodiments, the client computing device 175 (or other device, not shown) may additionally receive and use information about the generated mapping-related information, for example, to control or assist the automatic navigation activities of those devices (e.g., by autonomous vehicles or other devices), whether in lieu of or in addition to the visualization of the identified information.

[0028] In addition, Figure 1AIn the illustrated computing environment, computer network 170 can be one or more publicly accessible linked networks, possibly operated by various parties (such as the Internet). In other implementations, computer network 170 can take other forms. For example, computer network 170 can be replaced by a private network, such as a corporate or university network that is completely or partially inaccessible to unprivileged users. In other implementations, computer network 170 can include private and public networks, wherein one or more private networks are accessible and / or from one or more public networks. Furthermore, computer network 170 can include various types of wired and / or wireless networks in various situations. Additionally, client computing device 175 and server computing device 180 can include various hardware components and storage information, as referenced below. Figure 3 To be discussed in more detail.

[0029] Figure 1B A block diagram depicts an exemplary building interior environment in which various types of images have been captured and are ready for use (e.g., generating and providing corresponding enhanced images of the building and optional additional mapping information, and optionally also of the building's exterior and associated buildings (not shown, such as garages, sheds, ancillary living units, etc.)), as well as information for presenting the images and / or associated information to the user. Specifically, Figure 1BThe system includes a building 198 with an interior, which is at least partially captured by multiple images of various types, for example, by one or more users (not shown) carrying one or more devices with image acquisition capabilities to multiple acquisition locations 210 within the building. In this example, a first user may use a mobile image acquisition computing device 185 to capture a first set of one or more panoramic images at a first time, a second user may use a camera device 186 to capture a second set of one or more stereoscopic photographs at a separate second time, and one or more third users (not shown) may use one or more other devices (not shown) with visual data acquisition capabilities to capture a third set of one or more sets of additional visual data (e.g., one or more videos, additional panoramic images, additional stereoscopic photograph images, etc.) at one or more separate third times. As discussed elsewhere herein, the first set of images and optional additional information can be used to initially generate mapping information for the building, and can subsequently be updated to incorporate one or more additional images and optional additional information captured after the initial generation of the mapping information. In some embodiments, implementations of the AIC system (e.g., application 162 executed on the user's mobile image acquisition computing device 185) may automatically perform or assist in capturing data representing the building's interior. While the mobile image acquisition computing device 185 may include various hardware components, such as one or more cameras or other imaging systems 135, one or more sensor modules 148 (e.g., IMU gyroscope 148a, IMU accelerometer 148b, IMU compass 148c, etc., such as part of one or more IMUs or inertial measurement units of the mobile device; altimeter; light detector; etc.), GPS receiver, one or more hardware processors 132, memory and / or storage devices 152, display system 142, microphone, etc., in at least some embodiments, the mobile device may not have access to or use the equipment to measure the depth of an object in a building relative to the mobile device, making it possible to determine the relationship between different images and their acquisition location, partly or entirely based on matching visual elements in different images and / or by using information from other hardware components listed, but without using any data from any such depth sensor. Similarly, the camera device 186 may not have access to or use the equipment to measure the depth of an object in a building relative to the mobile device.In other embodiments, the mobile image acquisition computing device 185 and / or camera device 186 may optionally include and use one or more types of sensors and / or associated components to obtain depth data relating to structures (e.g., walls) and / or other objects in the environment surrounding the captured image, such as one or more depth sensing components 137 of the mobile image acquisition computing device 185 (e.g., using one or more technologies including LiDAR, structured light, time-of-flight, passive stereo, active stereo, etc.) corresponding to providing such data (optionally in conjunction with other components of the mobile image acquisition computing device 185). In at least some of these embodiments, such depth data may be used by the IAEMIGM system as part of generating enhanced images, as discussed in more detail elsewhere herein. Furthermore, in some embodiments, the mobile image acquisition computing device 185 and / or camera device 186 may optionally include and use one or more types of sensors and / or associated components to obtain location-related data of the acquired image and its surrounding environment, such as using sensor module 148 and / or imaging system 135 (e.g., using ARkit and / or ARCore or similar technologies), using other location-related components (e.g., a GPS receiver (not shown)), etc. In at least some of these embodiments, this location-related data can be used by the IAEMIGM system as part of the generation of enhanced images, as discussed in more detail elsewhere herein. Furthermore, although a direction indicator 109 is provided for the viewer's reference, in at least some embodiments, the (multiple) moving image acquisition computing devices 185 and / or (multiple) camera devices 186 and / or the AIC system may not use this absolute direction information, thus determining the relative direction and distance between images acquired at acquisition location 210 without considering the actual geographical location or orientation.

[0030] In operation, a user associated with the mobile image acquisition computing device 185 arrives at a first acquisition position 210A within a first room inside the building (in this example, the entrance passage from door 190-1 to the westernmost room (in this example, the living room)) and captures a view of a portion of the building's interior visible from that acquisition position 210A. In this example, the captured image is a 360° panoramic image that includes all or substantially all of the visual coverage 187a of the living room (and a small corridor to the east of the living room). The actions of the user and / or the mobile device can be controlled or facilitated by using one or more programs (e.g., application 162) executed on the mobile device, and the capture may include visual information depicting objects or other elements (e.g., structural details) that can be seen from the acquisition position in those directions. Figure 1BIn the example, such objects or other elements in building 198 include various elements that are structural parts of the walls (or "wall elements"), such as doorways 190 and 197 and their doors (e.g., swing and / or sliding doors), windows 196, and wall boundaries (e.g., corners or edges) 195 (including wall boundary 195-1 in the northwest corner of building 198, wall boundary 195-2 in the northeast corner of the living room, and wall boundary 195-3 in the southwest corner of the living room). Furthermore, in Figure 1B Such objects or other elements in the example may further include other elements within the room, such as a chaise lounge 191, a chair 192, a table 193, etc., pictures or paintings hanging on the wall, or a television or other objects 194 (e.g., pictures 194-1 and 194-2), lighting fixtures, etc. Users may also optionally provide textual or auditory identifiers associated with the acquisition location (e.g., the “entrance” or “living room” of acquisition location 210A), while in other embodiments, the IAEMIGM system may automatically generate such identifiers later (e.g., by automatically analyzing visual data of the building and / or other recorded information to perform the corresponding automatic determination, such as by using machine learning), or may not use identifiers at all.

[0031] After an image has been captured at the first acquisition location 210A, the user can move to another acquisition location (such as acquisition location 210D), optionally capturing motion data, such as visual data and / or other data from hardware components (e.g., from one or more IMUs, from a camera, etc.), during the movement between acquisition locations. At the next acquisition location, the user can similarly use the mobile device to capture one or more images from that acquisition location. This process can be repeated for some or all rooms of the building, and optionally outside the building, as shown for other acquisition locations 210B-210K. Furthermore, in this example, the user (whether the same or different) uses camera device 186 to capture one or more stereoscopic photographic images once or multiple times at one or more acquisition locations (whether the capture is the same or different from that of the mobile image acquisition computing device 185), including acquisition location 210C in this example, and optionally one or more of other acquisition locations 210B and 210E-210K. In this example, the perspective of the resulting stereoscopic photograph is the northwest portion of the living room, as shown by line of sight 187c. In this example, these sample images from acquisition locations 210A and 210C have an overlapping region 216ac, which can later be used by the IAEMIGM system to interconnect these images and / or their acquisition locations (for illustration, corresponding lines 215-AC are shown between them) to determine relative positional information between the two acquisition locations. (See also: Regarding...) Figure 2FAnd as discussed in more detail elsewhere in this article, and / or combining these images to generate one or more corresponding enhanced images, as discussed in [the context of...]. Figure 2D-Figure 2H For more detailed discussion. This automated operation of the IAEMIGM system can further generate and store other acquisition location pairs and / or corresponding interconnections of their captured images, and / or generate and store corresponding enhanced images, including, in some embodiments and cases, further connecting at least some acquisition locations whose images do not have overlapping visual coverage and / or are not visible to each other (e.g., the connection between acquisition locations 210E and 210K (not shown)).

[0032] about Figures 1A-1B Various details are provided, but it should be understood that the details provided are non-exclusive examples included for illustrative purposes, and other implementations may be carried out in other ways without some or all of these details.

[0033] Figures 2A-2J This demonstrates the automatic generation of mapping information for buildings using various types of images acquired by one or more devices at one or more times from one or more acquisition locations (including the generation of corresponding enhanced images, such as...). Figure 1B Examples of buildings discussed in 198, and the subsequent use of the generated mapping information in one or more automatic ways.

[0034] Specifically, Figure 2A It shows from Figure 1B An exemplary stereoscopic photograph 250a, taken from location 210B in the living room of building 198, is shown in a southwest direction. In this example, a direction indicator 109a is further shown to indicate the southwest direction of the image. In this example, a portion of windows 196-2 is visible, as is chaise lounge 191, and the visual horizontal and vertical room boundaries (including the horizontal boundary between the visible portion of the south wall of the living room and the ceiling and floor of the living room, the horizontal boundary between the visible portion of the west wall of the living room and the ceiling and floor of the living room, and the wall boundary 195-3 between the south and west walls). The example stereoscopic photograph 250a also shows the inter-room passageway of the living room, which in this example is the door 190-1 for entering and exiting the living room. Figure 1B (The sign indicates the door leading to the west exterior of the house).

[0035] Figure 2B continue Figure 2A Examples, and showed in Figure 1BAn additional stereoscopic photograph 250b, taken in the northwest direction from the living room of building 198, further shows a direction indicator 109b to indicate the northwest direction of the image taken, starting from acquisition position 210C. In the example shown, the image displayed includes built-in elements (e.g., light fixture 130b), portions of windows 196-1 and 196-2, and picture 194-1 hanging on the north wall of the living room. No room-to-room passageways (e.g., doors or other wall openings) are visible in this image. However, multiple room boundaries are shown in stereoscopic photograph 250a in a manner similar to... Figure 2A The manner is visible, including the wall boundary 195-1 between the north and west walls.

[0036] Figure 2C continue Figures 2A-2B Examples are shown, including panoramic image 250c with 360° visual coverage of a living room from acquisition location 210A, and visual data shown in an isorectangular format, in which horizontal lines in the living room (e.g., boundaries between walls and floor or ceiling) are shown with increasing curvature as their distance from the vertical center of the image increases. Vertical lines in the living room (e.g., boundaries between walls) are shown without such curvature. No direction indicators are shown in this example because the visual coverage of the panoramic image includes all horizontal directions. In the example shown, the image shows the same visual elements as in stereoscopic photographs 250a and 250b, as well as additional visual data not shown in those stereoscopic photographs (e.g., all windows 196-2 and 196-1, window 196-3, a portion of the hallway (including a portion of door 190-3) and corresponding wall openings on the east side of the living room, table 193, chair 192-1, ceiling light 130a, boundary between walls 195-2, etc.). It should be understood that various other types of structures and / or elements may exist in other embodiments.

[0037] Figure 2D continue Figures 2A-2CAn example is provided, and information 230d corresponding to the initial steps for determining the matching of stereoscopic photograph 250b with panoramic image 250c based at least in part on the overlapping visual overlay of the two images is shown. Specifically, in this example, multiple sub-image stereoscopic photographs 255a-255c are generated using panoramic image 250c. Sub-image stereoscopic photographs 255a-255c correspond to different subsets of the panoramic image and are converted to a stereoscopic planar format, such as sub-image 255c corresponding to the upward vertical direction (e.g., positive Z direction) from the acquisition position of the panoramic image, and sub-image stereoscopic photographs 255a and 255c corresponding to different horizontal directions (e.g., X or Y direction) from the acquisition position of the panoramic image (in this example, in the north and south directions). Although not shown in this example, additional such sub-images, such as additional stereoscopic photographs in the downward vertical direction, and additional stereoscopic photographs in other horizontal directions (e.g., in the east and west directions), can be generated similarly. It is understood that such sub-image stereoscopic photographs only show a subset of the visual data of panoramic image 250c.

[0038] Figure 2E and Figure 2F continue Figures 2A-2D Example, Figure 2E Information 230e is shown to provide a visual example of matching a subset of the stereoscopic photograph 250b with the sub-image stereoscopic photograph 255a of the panoramic image 250c, as shown in portion 220 superimposed on the sub-image stereoscopic photograph 255a. Figure 2F Additional information 230f is shown regarding the matching visual features between stereoscopic photograph 250b and sub-image stereoscopic photograph 255a. Specifically, Figure 2F This shows the visual data in two images (e.g., in...). Figure 1B Various types of matching features are visible in the overlapping region 216ac shown, including lines of sight 211a from acquisition position 210A and lines of sight 211c from acquisition position 210C, to determine the relative positions of those features from those acquisition positions (e.g., using a determined viewpoint orientation from each acquisition position to the matching feature to determine the relative rotation and translation between acquisition positions 210A and 210C, assuming sufficient overlap in the visual data of the two images). Figure 2F In the example, the matching features may include Figure 2FFeatures 229a, 229b, 229c, and 229d shown include, for example, the western edge or corner of window 196-1 of feature 229a; some or all of image 194-1 of feature 229b; one or more points in the center of the room of feature 229c (e.g., one or more points on the floor are visible and distinguishable from other points on the floor); and one or more points of feature 229d corresponding to the boundary between walls 195-1. It should be understood that many other features can be seen in both images, including points on structural elements (such as walls, floors, ceilings, windows, corners, boundaries, etc.) and points on non-structural elements (such as furniture). Some features may be visible only from one acquisition location, such as the boundary between walls 195-2, and therefore may not be used to compare and analyze visual data from images from these two acquisition locations (although it can be used to generate structural shapes from images captured at acquisition location 210A).

[0039] After analyzing multiple such features in the living room between acquisition locations 210A and 210C, various information regarding the positions of acquisition locations 210A and 210C within the room can be determined. Note that in this example, acquisition location 210C is near the boundary between the living room and the hallway and includes visual coverage of the living room (and such that directions different from this acquisition location can include visual data from multiple rooms, and thus can provide information about and be associated with one or two of these rooms). Similarly, the panoramic image acquired from acquisition location 210A can include visual data from a portion of the living room and the hallway. Although in this example the hallway can be modeled as a separate room, in other embodiments, such a hallway can alternatively be treated as part of one or more rooms connected to the hallway, or the hallway can alternatively be treated as a connecting passage between rooms rather than as a separate room. Similarly, small areas such as closets and / or niches / corners may not be analyzed as separate rooms, but rather treated as part of a larger accommodating room (optionally as unmapped spaces within the room), although in other embodiments, such small areas may alternatively be represented individually (including optionally having one or more acquisition locations located therein). While only the living room and two acquisition locations have been described, it should be understood that similar analyses can be performed for each acquisition location and for some or all of the rooms in a building, including optionally forming a larger image set including stereoscopic photograph 250a (e.g., based at least in part on matching visual features between stereoscopic photograph 250a and the sub-image stereoscopic photograph 255b generated from panoramic image 250c). Furthermore, analysis of the information in the images can be further used to determine additional location information for one or more such acquisition locations within a room, thereby further determining specific dimensions of the distance from the acquisition location to one or more nearby walls or other structural features of the room. In some embodiments, this dimensional determination information can be determined by using the dimensions of known objects (e.g., door frames, door handles, light bulbs, etc.) and the distances extrapolated to the corresponding dimensions and locations of other features. In this way, the analysis can provide the location of each acquisition point in the room, the location of the connecting passageways in the room, and optionally use structural information determined from one or more images to estimate the shape of part or the whole room, the visual data of which includes at least a portion of the room.

[0040] In some implementations, machine learning can be used to further perform automated determination of the location within a room and / or the estimated shape of all or part of the room, such as via a deep convolutional neural network that estimates the 2D or 3D layout of the room (e.g., rectangular or “box” shape; non-rectangular shape; etc.) from one or more images. This determination may include analyzing images to align visual data such that the floor is horizontal and the walls are vertical (e.g., by analyzing vanishing points in the images) and identifying and predicting corners and boundaries, where the resulting information fits into 2D and / or 3D forms (e.g., using layout parameters such as the outlines of the floor, ceiling, and walls to which the image information fits). Furthermore, in some implementations, humans can provide manual indications of the estimated room shape based on images, which can be used to generate corresponding floor plans and subsequently used to train models for use in the automatic generation of room shapes for other rooms from their images in subsequent scenarios. In some implementations, certain assumptions may be made to automatically analyze images of at least some rooms, such as one or more of the following: the room shape should be primarily rectangular / cuboid; if the room shape is not primarily rectangular / cuboid, multiple acquisition locations should be used within the room; the room should have at least a minimum number of visible corners and / or walls (e.g., 3, 4, etc.); the room should have a level floor and walls perpendicular to the floor; the walls should be flat rather than curved; images are acquired from a camera located at a specified level above the floor (e.g., 5 feet, approximately midway between the floor and ceiling, etc.); images are acquired from a camera at specified distances from one or more walls (e.g., 4 feet, 5 feet, 6 feet, 7 feet, 8 feet, 9 feet, 10 feet, etc.); etc. Furthermore, if multiple room shape estimates are available for the room (e.g., from multiple acquisition locations within the room), one can be selected for further use (e.g., based on the location of the acquisition location within the room, such as the most central location), or alternatively, multiple shape estimates can be optionally combined in a weighted manner. In at least some implementations, this automatic estimation of the room shape can be further performed by using one or more techniques, such as SfM (Structure from Motion), visual SLAM (Simultaneous Localization and Mapping), sensor fusion, etc. (if the relevant data is available).

[0041] Figure 2G and Figure 2H continue Figures 2A-2FExamples are provided, and example information 230g regarding possible types of attribute data exchange between the mapped stereoscopic photograph 250b and panoramic image 250c is shown. In this example, a sub-image stereoscopic photograph 255a of panoramic image 250c is shown instead of the entire panoramic image 250c, but attribute information exchange can occur between the entire panoramic image 250c instead of the shown sub-image stereoscopic photograph 255a. In this example, various chromaticity (or color) attribute data associated with stereoscopic photograph 250b are identified, and an example of an enhanced image that can be generated includes an enhanced panoramic image (not shown) in which the original panoramic image 250c is modified to use such chromaticity attribute data (e.g., only for portion 220 of panoramic image 250c, for all portions of panoramic image 250c corresponding to sub-image stereoscopic photograph 255a, for all of panoramic image 250c, etc.). Figure 2H Additional information 230h is shown to illustrate examples of generating different enhanced panoramic images by using color data from stereoscopic photographs with overlapping visual overlays.

[0042] also, Figure 2G Various structural information 255g can be determined from the panoramic image 250c in a manner discussed in more detail elsewhere, including structural information 239a that generates the 2D room shape of the living room in this example, the living room including the relative positions of walls and additional structural elements such as doors, windows, and openings between walls. The structural information 255g shown also illustrates that additional images with visual coverage of those rooms can be similarly used to determine other structural information for other rooms, such as the 2D structural shape 242b of a first bedroom adjacent to the southeast wall of the living room, the 2D structural shape 242a of another bedroom adjacent to the northeast wall of the living room, and the 2D structural shape 238b corresponding to a corridor, wherein the various structural shapes are optionally positioned relative to each other. Although not shown here, in some embodiments and situations, such structural shapes can be further represented in a 3D manner. Given this structural information 239a associated with the panoramic image 250c, another instance of an enhanced image generated from this pair of mapped images includes an enhanced stereoscopic photograph (not shown), wherein the original stereoscopic photograph 250b is modified to have some or all of the associated structural information data with structural information 239a. In this instance, a subset 239b of the structural information 239a corresponding to the visual data of the stereoscopic photograph 250b is determined and associated with the enhanced stereoscopic photograph, although in other embodiments, all of the structural information 239a may be associated with the enhanced stereoscopic photograph instead.

[0043] Furthermore, additional information can be determined, at least in part, based on stereoscopic photograph 250b, and optionally in combination with other stereoscopic photographs 250b, to learn features of a region of particular interest in building 198. Thus, as another example of one or more enhanced images that can be generated from this pair of survey images, such learned feature data can be used to analyze panoramic image 250c and select one or more subsets of panoramic images that match the learned features, wherein those selected subsets are used to generate one or more enhanced images (not shown, and whether generated as enhanced stereoscopic photographs or enhanced panoramic images), each enhanced image including visual data from one of those selected subsets. In other embodiments, as discussed in more detail elsewhere herein, a variety of other types of enhanced images can be generated in other ways.

[0044] In some non-exclusive exemplary implementations, regarding Figure 2D-Figure 2H The types of processing discussed may further include specific processing steps, as described below.

[0045] The first step in this example involves matching each stereo image with one of a plurality of panoramic images, each with 360° visual coverage in an isorectangular format, to find the visually most similar panoramic image (including finding the region in the panoramic image that is most similar to the stereo image). In this example, the matching includes isorectangular-to-stereo transformation, local and global feature extraction, and retrieval and pairing. First, each such panoramic image is divided into 6 non-overlapping stereo "crops" (or sub-images) to provide useful data for feature extraction and retrieval, as well as to facilitate finding the crop that is most similar to a given panoramic image. Next, depth features are extracted from each panoramic image crop and from each stereo image. Global features (having a single feature vector describing the entire corresponding cropped sub-image or stereo image) and local features (corresponding to a subset of the corresponding cropped sub-image or stereo image and having the extracted descriptors and their corresponding 2D positions in the corresponding cropped sub-image or stereo image) help to effectively narrow down the candidates for the most similar panoramic image, while local features are used to select the most similar panoramic image from the candidates, including performing geometric verification of the panoramic image and the stereo image. The next step in this example involves pairing each stereo image with its most similar panoramic image using both global and local features. Specifically, for each stereo image, its global features are used to find the panoramic image most similar to a given stereo image. Local features are then used to geometrically validate, for example, the top 10 retrieved panoramic images and reorder them based on the degree of matching (i.e., visual similarity). Then, based on the number of inliers in the geometric validation step, a cropped "best" panoramic image for each stereo image (i.e., the highest-ranking cropped sub-image with the highest local feature matching) is associated with the stereo image.

[0046] To perform the transfer and exchange of chromaticity attribute data and / or other photometric attribute data (e.g., color profile, exposure, etc.) between two images in a pair, one of the images in the pair (e.g., a panoramic image) is modified to use such photometric attribute data from the other image in the pair (e.g., a stereograph). In various implementations, this photometric attribute data exchange can be performed in various ways; in some implementations, the algorithm described by Reinhard et al., “Color Transfer between Images”, IEEE Computer Graphics and Applications, 2001, is used, and optionally, this algorithm or similar algorithms are modified to allow the amount of transfer in the luminance channel to be customizable (e.g., set to zero so that only chromaticity or ab values ​​are transferred). For color profile transfer from stereographs to enhance panoramicity, stereographs are typically captured and edited professionally to give them visual appeal, where this chromaticity / color profile attribute data exchange is used to generate an enhanced panoramic image with higher-quality visual data (and correspondingly higher-quality generated mapping information).

[0047] To perform a transfer exchange of structural attribute data (e.g., structural shape, position of structural elements, relative arrangement of multiple structural pieces, etc.) between two images in a pair, one of the images in the pair (e.g., a stereoscopic photograph) is modified to use such structural attribute data from the other image in the pair (e.g., a panoramic image). In some implementations, this involves matching coordinate data between the two images so that it begins with structural attribute data in a local coordinate system used by the panoramic image, begins with a separate global coordinate system (or a different local coordinate system used by the stereoscopic photograph), begins with information indicating or used to determine the position of the panoramic image in the global coordinate system (or the different local coordinate system of the stereoscopic photograph), and optionally begins with information indicating or used to determine the position of the stereoscopic photograph in the global coordinate system. If such a position of the stereoscopic photograph is also available, it can be used to locate the structural data from the panoramic image into the coordinate system used by the stereoscopic photograph. The method includes the following steps:

[0048] - For each set of structural data generated from the corresponding panoramic image or otherwise obtained, the center of the panoramic image is represented as the center of the local coordinate system of the particular room shape.

[0049] - Transform the room shape from the local coordinate system of the panoramic image to the global coordinate system, and then back to the local coordinate system of the stereoscopic image (or, in other implementations, directly to the local coordinate system of the stereoscopic image). If the location of the stereoscopic image is not provided, an algorithm such as SfM or any other pose estimation method can be used to estimate it relative to the panoramic image, and then the structural data of the panoramic image of the stereoscopic image can be located based on that location.

[0050] To perform attribute data transfer corresponding to characteristics of a house or other building of interest using two images in a pair, one of the images in the pair (e.g., a panoramic image) is modified, at least in part, based on the other image in the pair (e.g., a stereoscopic photograph), according to the identified characteristics. For example, stereoscopic photographs are typically captured to reflect the “best” or other preferred viewpoint and / or angle for a particular room and / or building, and corresponding features can be identified from the stereoscopic photograph and used to select a corresponding subset of one or more panoramic images to learn how to create cropped sub-images of panoramic images with the corresponding “best” or other preferred viewpoint and / or angle for a particular room and / or building. In some implementations, for a set of stereoscopic photographs and panoramic images corresponding to multiple other buildings, one or more subsets of panoramic images matching one or more stereoscopic photographs are used as positive samples for training a corresponding machine learning model, and other regions of the matching panoramic images are used as negative samples for such training. Given the positive and negative samples, a machine learning model can be trained to automatically select corresponding subset views within other panoramic images and create corresponding enhanced images based on cropped sub-images of these views. Furthermore, additives from Generative Advertising Networks (GANs) and / or similar techniques can be used to match the style of the selected panoramic image view with a stereoscopic photograph view (e.g., the style of a stereoscopic photograph in a pair that matches the panoramic image), thereby enhancing the quality of the resulting augmented image. In some cases, this technique can enable the generation of the same type of mapping information for several buildings based solely on a set of panoramic images of those buildings, without requiring any stereoscopic photographs of those buildings.

[0051] Figure 2I Information 290i is also shown to illustrate exemplary architectures and information processing flows for performing some or all of the described techniques. In this example, one or more computing devices 188 perform... Figure 1A The implementation of the IAEMIGM system 160, and optionally execution Figure 1A Application 162 (and / or Figure 3 The implementation of the AIC system application 368) is adapted to correspond to Figure 1A The server computing device 180 and optionally the mobile image acquisition computing device 185. Figure 2IAn implementation of the IAEMIGM system 160 performs the following steps: process 283, receiving multiple types of images of one or more related buildings; various processing steps 284 of the visual data of the images; process 286, generating pairs of related images of different types; process 287, exchanging attribute data between the image pairs to generate enhanced images; optionally, process 288, using the enhanced images to generate improved mapping information; and process 289, displaying or otherwise providing the enhanced images of the buildings and / or other generated mapping information. The images received in process 283 may come from, for example, image storage location 295 (e.g., a database) and / or from the execution of the AIC system. If the AIC system application 368 provides some or all of the images, it can perform processing 281 to capture and receive images using one or more imaging systems 135, and optionally receive additional information from sensor module 148, and optionally perform processing 282 to select a subset of images to use (e.g., based on properties of those images, such as sharpness of sufficient detail or other indicators in the visual data of those images for further analysis) and / or modify the images in one or more ways (e.g., change the format of the images, crop or resize them, or otherwise modify the images). As part of processing 284 of the visual data of the images, the IAEMIGM system 160 can perform processing 285a to estimate the pose of the images, and perform processing 285b to determine the features visible in the visual data of the images, such as in a manner discussed elsewhere in this document in more detail. Process 284 may also include one or more additional steps related to determining attribute information associated with a particular image, in order to perform: process 285c, to determine structural information from the visual data of the image; process 285d, to determine color information from the visual data of the image; and / or process 285e, to identify regions of interest in the image and learn the corresponding characteristics of these regions, such as in a manner discussed elsewhere in detail herein. Process 284 may also optionally perform additional activities to perform: process 285f, to determine one or more other types of attribute information based on the visual data and / or other information associated with the image. Different types of related image pairs may then be determined in various ways, such as based at least in part on the visual overlap of these images and / or using other criteria, using information from one or more of processes 285a and / or 285b, and in a manner discussed elsewhere in detail herein. Similarly, one or more types of attribute data can be exchanged between a pair of images to generate one or more enhanced images, using information from processing 285c and / or 285d and / or 285e and / or 285f, and in a manner discussed in more detail elsewhere in this document.In the illustrated embodiment, one or more computing devices 188 may also interact with one or more other systems via one or more computer networks 170, including one or more server computing devices 180, on which information generated by the IAEMIGM system may be stored for later use.

[0052] Figure 2J continue Figures 2A-2H The example illustrates a 2D floor plan 235j of building 198, which can be presented to an end user in a GUI, where the living room is the westernmost room of the house (as reflected by the direction indicator 209). It will be understood that in some implementations, 3D or 2.5D computer models with wall height information can be similarly generated and displayed, either as an addition to or replacement of such a 2D floor plan. In this example, various types of information are shown on the 2D floor plan 235j. For example, this type of information may include one or more of the following: room labels added to some or all rooms (e.g., “living room” for a living room); room dimensions added to some or all rooms; visual indications of fixtures or appliances or other built-in features added to some or all rooms; visual indications added to some or all rooms for the location of additional types of associated and linked information (e.g., panoramic images and / or stereoscopic photographs and / or videos and / or other visual data optional for further display by the end user, audio annotations and / or sound recordings optional for further presentation by the end user, etc.); visual indications added to some or all rooms for doors and windows; built-in features (e.g.,... Visual indicators for (e.g., kitchen island); visual indicators for installed fixtures and / or appliances (e.g., kitchen appliances, bathroom items, etc.); visual indicators for appearance and other surface information (e.g., the color and / or material type and / or texture of installed items, such as floor coverings or wall coverings or surface coverings); visual indicators for views from a particular window or other building location and / or visual indicators for other information about the exterior of the building (e.g., the type of exterior space; items present in the exterior space; other related buildings or structures, such as sheds, garages, pools, platforms, patios, walkways, gardens, etc.); keywords or text descriptions 269 for identifying visual indicators for one or more types of information; etc. When displayed as part of the GUI, some or all of such displayed information may be user-selectable controls (or associated with such controls) that allow the end user to select and display some or all of the associated information (e.g., selecting a 360° panoramic image indicator for acquisition location 210A or a stereoscopic photograph indicator for acquisition location 210B or 210C to view some or all of the corresponding panoramic image or stereoscopic photograph (e.g., in a similar manner to...). Figures 2A-2C(In this example, a user-selectable control 228 is added to indicate the current floor displayed on the floor plan and to allow the end user to select a different floor to be displayed.) In some embodiments, changes to floors or other levels can also be made directly from the floor plan, such as by selecting the corresponding connecting passage in the floor plan (e.g., a staircase to floor 2). It will be understood that various other types of information may be added in some embodiments, some of the information of the types shown may not be provided in some embodiments, and in other embodiments, visual indications and user selections of linked and associated information may be displayed and selected in other ways.

[0053] Furthermore, as described elsewhere herein, in some embodiments, one or more corresponding GUIs may be provided to enable user input to supplement information automatically determined by the IAEMIGM system, such as for connecting disjoint subgraphs, editing / removing connections found incorrect by the user, adjusting relative positions, or other automatically determined information. Additionally, in at least some embodiments, the user experience can be enhanced by personalizing the user's visualization based on user-specific information (e.g., user history, user-specified preferences, etc.) and / or machine learning predictions. Furthermore, in some embodiments, crowdsourced improvements for the entire system for other users may be used, such as by obtaining and incorporating additional images and / or other visual data based on edits from each individual user (e.g., after the IAEMIGM system performs initial automatic determination using an initial image set and optional additional information). It will be understood that additional types of user-selectable controls may be used.

[0054] Already referred to Figures 2A-2J Various details are provided, but it should be understood that the details provided are non-exclusive examples included for illustrative purposes, and other implementations may be carried out in other ways without some or all of these details.

[0055] Figure 3This is a block diagram illustrating an implementation of one or more server computing systems 300 performing an implementation of the IAEMIGM system 340. The server computing systems and the IAEMIGM system can be implemented using multiple hardware components forming electronic circuitry adapted and configured to perform at least some of the techniques described herein in combined operation. In the illustrated embodiment, each server computing system 300 includes one or more hardware central processing units (“CPUs”) or other hardware processors 305, various input / output (“I / O”) components 310, storage devices 320, and memories 330. The illustrated I / O components include a display 311, a network connection 312, a computer-readable media driver 313, and other I / O devices 315 (e.g., a keyboard, mouse, or other pointing device, microphone, speaker, GPS receiver, etc.).

[0056] The server computing system 300 and the IAEMIGM execution system 340 can communicate with other computing systems and devices via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.), such as user client computing devices 390 (e.g., for displaying or otherwise using the generated enhanced images and / or other mapping-related information, and optionally associated images or other visual data), mobile image acquisition computing devices 360 (e.g., on which AIC system applications 368 may optionally be executed to perform the acquisition of capture images 366), and optionally, if the camera device includes networking capabilities or other data transmission capabilities, capture. One or more camera devices 385 that acquire images 386, optionally other storage devices 380 (e.g., for storing and providing additional information related to buildings; for storing and providing captured images, such as in place of the mobile image acquisition computing device 360 ​​and / or camera device 385 or in addition to the mobile image acquisition computing device 360 ​​and / or camera device 385; for storing and providing generated enhanced images and / or other generated building mapping information; etc.), and optionally other navigable devices 395 that receive and use the generated mapping-related information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices).

[0057] In the illustrated embodiment, the implementation of the IAEMIGM system 340 is executed in memory 330 to perform at least some of the described techniques, such as executing software instructions of the IAEMIGM system 340 in a manner that configures the hardware processor 305 and the server computing system 300 to perform automated operations that implement those described techniques. The illustrated implementation of the IAEMIGM system may include one or more components (not shown) to each perform a portion of the functionality of the IAEMIGM system, and the memory may further optionally execute one or more other programs 335. As a specific example, in at least some embodiments, a copy of the AIC system may be executed as one of the other programs 335, such as in place of or in addition to the AIC system application 368 on the mobile image acquisition computing device 360, and as another specific example, a copy of the building information viewer system 345 may be executed to provide a visual representation of the generated enhanced images and / or other generated building mapping information to an end user (e.g., a user of the client computing device 390), although in other embodiments, such a mapping information viewer system may alternatively be executed on one or more such client computing devices 390. The IAEMIGM system 340 can also store and / or retrieve various types of data on storage device 320 during its operation (e.g., in one or more databases or other data structures), such as various types of user information 322, acquired images and other associated information 324 (e.g., image acquisition IMU data and / or other metadata, and such as for analysis to generate enhanced images and / or other building mapping related information; to provide a display to the user of client computing device 390; etc.), one or more types of image attribute data 325 of the acquired images, generated enhanced images 326, optional other generated mapping related information 327 (e.g., generating sub-maps for use with associated floor plans, linking acquisition locations, floor plans and / or component structural shapes, building and room dimensions, etc.) and / or various types of optional additional information 329 (e.g., additional images and / or video and / or other visual data), non-visual data captured in the building, annotation information, various analytical information related to the presentation or other use of one or more building interiors or other environments captured by the AIC system, etc.

[0058] Some or all of the user client computing device 390 (e.g., a mobile device), mobile image acquisition computing device 360, camera device 385, storage device 380, and other navigable devices 395 may similarly include some or all of the same type of components shown for the server computing system 300. As a non-limiting example, each of the mobile image acquisition computing devices 360 is shown as including one or more hardware CPUs 361, I / O components 362, memory and / or storage devices 367, and an imaging system 365 and an IMU hardware sensor 363, and having an implementation of an AIC system application 368 on the memory / storage device 367 and captured images 366 generated by the AIC system, as well as optional other programs (such as a browser 369). While specific components for other navigable devices 395 or other storage devices 380 and camera devices 385 and client computing devices 390 are not shown, it should be understood that they may include similar and / or additional components.

[0059] It should also be understood that, including Figure 3 The server computing system 300 and other systems and devices described herein are merely illustrative and not intended to limit the scope of the invention. Systems and / or devices may instead comprise multiple interactive computing systems or devices and may connect to other devices not specifically shown, including via Bluetooth communication or other direct communication, via one or more networks such as the Internet, via a web page, or via one or more dedicated networks (e.g., mobile communication networks, etc.). More generally, devices or other computing systems may include any combination of hardware capable of interacting and performing functions of the type described, optionally when programmed or otherwise configured with specific software instructions and / or data structures, including but not limited to desktop computers or other computers (e.g., input boards, tablets, etc.), database servers, network storage devices and other network devices, smartphones and other cellular phones, consumer electronics devices, wearable devices, digital music player devices, handheld gaming devices, PDAs, wireless phones, Internet devices, and various other consumer products including suitable communication capabilities. Furthermore, in some embodiments, the functions provided by the illustrated IAEMIGM system 340 may be distributed across various components, some of the functions described in the IAEMIGM system 340 may not be provided, and / or other additional functions may be provided.

[0060] It should also be understood that although various items are shown as stored in memory or storage devices during use, these items or portions thereof may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software components and / or system may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Thus, in some embodiments, when configured by one or more software programs (e.g., by the IAEMIGM system 340 and / or AIC system application 368 executed on server computing system 300 and / or mobile image acquisition computing device 360) and / or data structures, some or all of the described techniques may be executed by hardware devices including one or more processors and / or memory and / or storage devices, such as by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures, and thereby executing algorithms as described in the flowcharts and other disclosures herein. Furthermore, in some implementations, some or all of the system and / or components may be implemented or provided in other ways, such as by means of one or more means implemented in firmware and / or hardware (e.g., rather than means implemented wholly or partially by software instructions configuring a particular CPU or other processor), including but not limited to one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on non-transitory computer-readable storage media, such as hard disks or flash drives or other non-volatile storage devices, volatile or non-volatile memories (e.g., RAM or flash RAM), network storage devices, or portable media articles (e.g., DVDs, CDs, optical discs, flash memory devices, etc.), which will be read by an appropriate drive or via an appropriate connection. In some embodiments, the system, components, and data structures can also be transmitted over various computer-readable transmission media (including wireless-based and wired / cable-based media) via generated data signals (e.g., as part of a carrier or other analog or digital propagation signal), and can take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such a computer program product can also take other forms. Therefore, embodiments of this disclosure can be implemented using other computer system configurations.

[0061] Figures 4A-4BAn exemplary implementation of a flowchart for an Image Attribute Exchange and Mapping Information Generation Manager (IAEMIGM) system routine 400 is shown. For example, it can be implemented by executing... Figure 1A IAEMIGM system 160, Figure 3 The IAEMIGM system 340, and / or as referenced Figure 2D-Figure 2J This routine is executed using the IAEMIGM system described elsewhere in this document to generate, at least in part, enhanced images of defined areas and / or other mapping-related information (e.g., linked acquisition locations and associated images, at least partial floor plans, etc.) based on images of various types of areas. Figures 4A-4B In the example, the generated mapping information includes enhanced images, linked acquisition locations and associated images, and at least partial 2D floor plans and 3D computer models of buildings (such as houses). However, in other implementations, other types of mapping information can be identified and generated for other types of buildings and used in other ways, as discussed elsewhere herein.

[0062] The illustrated implementation of the routine begins at box 405, where information or instructions are received. The routine continues to box 407 to determine whether the instructions received in box 405 instruct the current acquisition of various types of images at multiple acquisition locations within the indicated building, and optionally, the acquisition of other relevant data. If so, the routine proceeds to box 409 to do so (optionally waiting for one or more users or devices to move within the building and acquire images at multiple acquisition locations in multiple rooms of the building, and optionally, acquire additional information, as discussed in more detail elsewhere herein). Figure 7 An exemplary implementation of an AIC system routine for performing at least some of such image acquisition is provided, and otherwise proceeds to box 410 to obtain multiple types of images of multiple acquisition locations of a previously acquired building, as well as optionally other relevant data. After box 409 or 410, the routine continues to box 415 to determine whether the instructions or information received in box 405 instruct the performance of attribute determination and / or exchange of multiple types of building images (including generating one or more enhanced images based on such attribute data exchange), and if so, proceeds to box 420 to do so, wherein... Figure 5 An example of a routine for performing such an activity is shown.

[0063] Following box 420, or if it is determined in box 415 that the instructions or other information received in box 405 do not require attribute determination or exchange, the routine continues to box 430 to determine whether the instructions received in box 405 instruct the generation of other types of mapping information for the indicated building. If so, the routine continues to execute boxes 435-488 to do so; otherwise, it continues to box 480. Specifically, in box 435, in addition to the images from boxes 409 or 410, the routine optionally obtains additional information about the building, optionally along with metadata information relating to image acquisition and / or movement between acquisition locations, such as information that may in some cases have already been provided in box 405 along with the corresponding instructions. This additional information may, for example, be based on acquired annotation information and / or information from one or more external sources (e.g., online databases, information provided by one or more end users, etc.) and / or information from the analysis of acquired images (e.g., initial images and / or supplementary images, such as supplementary images captured at locations different from the acquisition location of the initial images). Such additional information may include, for example, the building’s external dimensions and / or shape; information about built-in features (e.g., kitchen island); information about installed fixtures and / or appliances (e.g., kitchen appliances, bathroom items, etc.); visual appearance information about interior locations of the building (e.g., the color and / or material type and / or texture of installed items such as floor coverings or wall coverings or surface coverings); information about views from specific windows or other building locations; and other information about exterior areas of the building (e.g., other related buildings or structures such as sheds, garages, pools, platforms, courtyards, walkways, gardens, etc.; the type of exterior space; items present in the exterior space; etc.).

[0064] Following box 435, the routine continues to box 445 to obtain room structure shape data generated from images of the building (e.g., as part of box 420), or otherwise determine such room structure data. For example, for each room within a building having one or more acquisition locations and associated acquisition images, the room shape may be determined, for instance, automatically, based on data from images taken within the room and optionally at a specified location within its acquisition location within the room. The operation of box 445 may further include using visual data from the images and / or acquisition metadata of the images to determine, for each room in the building, any connecting passageways for entering or leaving the room (e.g., automatically), and any wall elements within the room and their locations (e.g., automatically), such as windows, boundaries between walls, etc. The operation of box 445 may also include using some or all of the other information determined in box 445 to determine an estimated room shape. The routine then proceeds to box 455, where it uses the determined room shapes to create an initial 2D floor plan, such as by connecting the passageways between the rooms in their respective rooms, by optionally positioning the room shapes around the determined acquisition locations of the image (e.g., if the acquisition locations are interconnected), and by optionally applying one or more constraints or optimizations. This floor plan may include, for example, the relative positions and shapes of various rooms without providing any actual size information for the individual rooms or the building as a whole, and may also include multiple linked or associated sub-maps of the building (e.g., to reflect different floors, levels, sections, etc.). The routine also associates the positions of doors, wall openings, and other identified wall elements on the floor plan.

[0065] Following box 455, the routine optionally executes one or more boxes 460-470 to determine additional information and associate that information with the floor plan. In box 460, the routine optionally estimates some or all of the dimensions in the rooms, such as from analysis of images and / or their acquisition metadata or from overall dimensional information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan. It should be understood that if sufficiently detailed dimensional information is available, architectural drawings, blueprints, etc., can be generated from the floor plan. Following box 460, the routine continues to box 465 to optionally associate further information with the floor plan (e.g., with specific rooms or other locations within the building), such as additional existing images with specified location and / or annotation information. In box 470, the routine further estimates the heights of the walls in some or all of the rooms, such as from analysis of the images and optionally from dimensions of objects known in the images, and height information about the camera when the images were acquired, and further uses this information to generate a 3D computer model floor plan of the building, where the 2D and 3D floor plans are associated with each other. Although not shown here, any information determined with respect to boxes 445-470 can be similarly associated with the correspondingly generated enhanced image (e.g., the enhanced image generated in box 420). Additional details of exemplary implementations of the operation of boxes 435-470 are included elsewhere herein. Furthermore, it should be understood that although boxes 435-470 are shown as separate operations in this example, in some implementations, a single analysis of the image can be performed to acquire or determine multiple types of information, such as those discussed in some or all of boxes 435-470. Following box 470, the routine continues to box 488 to store the generated mapping information and optionally other generated information, and optionally further use the generated mapping information to provide the generated mapping-related information to one or more recipients (e.g., to provide the generated information to other devices for use in the automated navigation of these devices and / or associated vehicles or other entities).

[0066] If it is determined in box 430 that the information or instruction received in box 405 is not intended to generate mapping information for the indicated building, the routine continues to box 490 to optionally perform one or more other instructed operations, as appropriate. Such other operations may include, for example, receiving and responding to requests for previously generated computer models and / or floor plans and / or generated enhanced images and / or other generated mapping-related information (e.g., requests for such information to be provided to one or more other devices for use in automated navigation; requests to display such information on one or more client devices to match such information against one or more specified search criteria; etc.), obtaining and storing information about the building to be used in subsequent operations (e.g., information about room size, number or type, total square feet, other buildings nearby, nearby vegetation, exterior images, overhead images, street-level images, etc.), etc.

[0067] After box 488 or 490, the routine continues to box 495 to determine whether to continue, such as until an explicit instruction to terminate is received, or only if an explicit instruction to continue is received. If it is determined to continue, the routine returns to box 405 to wait for and receive additional instructions or information; otherwise, it continues to box 499 and terminates.

[0068] Figure 5 An exemplary implementation of the IAEMIGM image attribute exchange component routine 500 is shown, illustrated in the flowchart. For example, it can be implemented by executing... Figure 1A Components of the IAEMIGM system 160 Figure 3 Components of the IAEMIGM system 340, and / or as referenced Figure 2D-Figure 2J The components of the IAEMIGM system, as described elsewhere in this document, are used to execute this routine in order to determine and exchange attribute data between various types of images of a building in order to generate corresponding enhanced images. In at least some embodiments, it is possible to... Figures 4A-4B The routine 400 is called at box 420 and returns to that location upon completion. While the embodiment of the routine 500 shown uses specific type of attribute data shared between specific types of images paired together for a specific building to generate a specific type of enhanced image, it should be understood that other embodiments of the routine may operate in other ways, including using other types of attribute data and / or image types, and / or using groups of more than two matching images, and also including doing so for multiple buildings (e.g., multiple related buildings, such as on a single property; multiple individual buildings being compared; etc.).

[0069] The illustrated implementation of routine 500 begins at block 505, where instructions or other information, such as regarding a specific building and / or a specific set of images of various types acquired for that specific building, are received. The routine then proceeds to block 510, where it is determined whether the instructions or other information received in block 505 instructs analysis of the images of the specific building to determine their attribute information. If so, blocks 520-535 are executed to determine the corresponding attribute information; otherwise, the routine proceeds to block 515 to retrieve or otherwise obtain stored attribute information of one or more types for such building images. In block 520, the routine determines the color attributes and optionally other visual attributes of the visual data for each image. In block 525, the routine then analyzes the visual data of the images to determine structural data of information visible in the images for each image, including features corresponding to wall elements and other structural elements, and to estimate the pose (acquisition location and capture orientation) for each image, and optionally further determine corresponding semantic labels and associate the corresponding semantic labels with the identified features. In box 530, the routine then correlates the images by room in the building, and for each room having one or more image acquisition locations (or additionally visual coverage of at least some rooms in the images), determines the structural shape of at least some rooms based at least in part on the visual data of those images, and optionally further determines the location within the rooms of those acquisition locations (e.g., location relative to the structural shape) and / or further determines the corresponding room type semantic labels for some or all rooms, and correlates the determined structural data and other information from boxes 530 and 535 as attributes of the corresponding images. In box 535, the routine then optionally analyzes the visual data of each image to generate an embedding vector representing information about it, such as for comparing image embedding vectors to determine similar images.

[0070] Following box 515 or 535, the routine continues to box 545, where it determines whether the instructions or other information received in box 505 instructs the performance of attribute exchange between images, such as regarding attribute data retrieved in box 515 or determined in boxes 520-535. If not, the routine continues to box 599 and returns (including providing any determined and generated information from the routine); otherwise, it proceeds to boxes 555-570 to perform the attribute exchange. Specifically, in box 555, the routine identifies different types of image pairs with overlapping visual coverage, and in box 560, optionally, one image (e.g., a stereoscopic photographic image) with preferred color attribute data and optionally other visual data attributes is selected from each image pair, and the other image in the pair (e.g., a panoramic image) is modified to generate an enhanced image using those color attributes and optionally other visual data attributes. In box 565, then, for each image pair, the routine optionally selects one image (e.g., a panoramic image) with preferred structural attribute data and modifies the other image in the pair (e.g., a stereoscopic photograph) to generate a corresponding augmented image associated with the structural attribute data (in at least some implementations and cases, this includes adapting the structural data to the local coordinate system of the other image in the pair). In box 570, then, for each image pair, the routine optionally selects an image with a larger size or an image with the largest visual coverage (e.g., a panoramic image), obtains information about identified characteristics of the visual data of particular interest (e.g., analyzing the other image in the pair, such as a stereoscopic photograph, and / or other images to learn the characteristics), and generates one or more augmented images from the selected image using the identified characteristics, each augmented image corresponding to a subset and representing an augmented image of particular interest. After box 570, the routine returns to box 599, including providing any determined and generated information from the routine.

[0071] Figure 6 An exemplary implementation of a flowchart for a building information viewer system routine 600 is shown. This routine can be implemented, for example, by executing... Figure 1A Building information viewer user client computing device 175 and its software system (not shown) Figure 3The building information viewer system 345 and / or client computing device 390, and / or a mapping information viewer or presentation system as described elsewhere herein, are used to perform the following: select information for one or more buildings based on user-specific criteria (e.g., one or more generated enhanced images of a building; a group of one or more interconnected linked images, each representing some or all of the building; one or more 2D floor plans of the building and / or other relevant mapping information, such as 3D or 2.5D computer models; additional information or other mapping information associated with specific locations in the floor plans, such as additional images acquired within the building and / or other types of data; etc.), and receive and display the corresponding building information to the end user. Figure 6 In the example presented, the mapping information is used at least in part for the interior of a building (such as a house), but in other implementations, other types of mapping information may be presented for other types of buildings or environments and used in other ways, as discussed elsewhere in this document.

[0072] The illustrated implementation of the routine begins at box 605, where an instruction or information is received. Following box 605, the routine continues to box 650, where it determines whether the instruction or other information received in box 605 indicates identifying one or more target buildings to be presented based at least in part on user-specific criteria. If not, it continues to box 660 to obtain an indication from the end user of the target building to be used (e.g., based on the current user selection, such as from a displayed list or other user selection mechanism; based on the information received in box 605; etc.). Otherwise, the routine continues to box 655, where it obtains an indication of one or more search criteria to be used (one or more initial buildings for identifying similar target buildings, such as from the current user selection and / or from previous user selections or other previous user activities and / or as indicated in the information or instruction received in box 605; one or more explicit search terms; etc.). The routine then obtains information about one or more corresponding target buildings, such as by requesting that information from the IAEMIGM system and / or associated storage system, and if information about multiple target buildings is returned, selects the best-matching target building for initial further use (e.g., other buildings returned with the highest similarity level to one or more initial buildings, or for one or more specified search criteria, or another selection technique indicated in instructions or other information previously specified in box 605 or otherwise, such as end-user preferences). In some implementations and situations, one or more target buildings may be selected based on one or more other buildings and one or more specified criteria.

[0073] Following box 655 or 660, the routine continues to box 670 to determine whether the instruction or other information received in box 605 should display one or more generated augmented images, such as the target building indicated by the best matching target building from box 655 or other indications in box 660. If so, it continues to box 675 to obtain one or more indications of one or more corresponding augmented images (e.g., some or all augmented images associated with the target building) and to display or otherwise provide information about the augmented images to one or more users (e.g., providing information about multiple available augmented images and displaying one or more of such augmented images when the user selects or otherwise indicates it). Although not shown in this exemplary embodiment, in some embodiments, the displayed augmented images may include one or more user-selectable controls corresponding to available association information, and if so, the routine may also perform additional processing in a manner similar to that discussed for boxes 615-622 to enable the user to further interact with and receive additional association information.

[0074] Following box 675, or if instead it is determined in box 670 that the instructions or other information received in box 605 do not display one or more generated enhanced images, the routine continues to box 610 to determine whether the instructions or other information received in box 605 should display or otherwise present other types of information about the target building (e.g., using a set of floor plans and / or interconnected images that include information about the interior of the target building), such as the best-matching target building from box 655 or other indicated target buildings from box 660, and if not, continues to box 690. Otherwise, the routine proceeds to box 612 to retrieve other building information about the target building (optionally including indications of associated or linked information about the interior and / or surrounding locations of the building, such as additional images taken inside or around the building), and selects an initial view of the retrieved information (e.g., a view of a floor plan, a view of at least some of the 3D computer models, images from a set of interconnected images, a visualization of multiple linked images, etc.). In box 615, the routine then displays or otherwise presents the current view of the retrieved information and waits for user selection in box 617. After the user selection in box 617, if it is determined in box 620 that the user selection corresponds to the current target building location (e.g., changing the current view of the displayed mapping information of the target building), the routine proceeds to box 622 to update the current view according to the user selection, and then returns to box 615 to update the displayed or otherwise presented information accordingly. User selection and corresponding updates to the current view may include, for example, displaying or otherwise presenting an associated link of information selected by the user (e.g., a specific image associated with a visual indication displayed at a determined acquisition location), changing how the current view is displayed (e.g., zooming in or out; rotating information if appropriate; selecting new portions of the floor plan and / or 3D computer model to be displayed or otherwise presented, such as some or all of previously invisible new portions, or instead, a subset of previously visible information; selecting different images from a set of interconnected linked images to be displayed or otherwise presented, so as to display an initial subset view of that image; etc.). In other implementations and situations, updating the current view based on user selection may include interacting with another system and retrieving information from another system (such as from an IAEMIGM system).

[0075] If it is determined in box 610 that the instructions or other information received in box 605 are not intended to present information representing the interior of the building, the routine continues to box 690 to perform any other instructed operations as appropriate, such as any household chores, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by the user of the system, such as the user capturing one or more mobile devices inside the building, the operator user of the AIC system, etc.), to obtain and store other information about the user of the routine (e.g., the presentation and / or search preferences of the current user), to respond to requests for information generated and stored, etc.

[0076] After box 690, or if it is determined in box 620 that the user's selection does not correspond to the current target building location, the routine proceeds to box 695 to determine whether to continue, such as until an explicit instruction to terminate is received, or only if an explicit instruction to continue is received. If it is determined to continue (including in the case where the user makes a selection related to a new target building location to be presented in box 617), the routine returns to box 605 to await additional instructions or information (or if the user makes a selection related to a new building location to be presented in box 617, it continues through boxes 605 and 650 to boxes 670 or 610), and terminates if no further steps are reached at step 699. In the illustrated embodiment, if multiple target building candidates return to the block, the routine in box 655 selects the best matching target building to use. In at least some implementations, a queue of other such returned target buildings that were not initially selected as the best match may be further saved and subsequently used (e.g., for the user, to be continuously displayed or otherwise presented for multiple such other buildings) such as the user selection in box 617 optionally indicating to select and use the next returned other building from such queue, and / or information about multiple buildings may be displayed together (e.g., simultaneously or sequentially so that information about multiple buildings can be compared).

[0077] Figure 7 An example flowchart illustrating an implementation of an Automatic Image Capture (AIC) system routine 700 is shown. This routine can be, for example, provided by... Figure 3 AIC system application 368, Figure 1A Application 162, and / or as about Figures 2A-2CThis is performed using the AIC system described elsewhere in this document to capture one or more types of images at acquisition locations within a building or other structure, such as for subsequent generation of related floor plans and / or other mapping information. While the section discussing example routine 700 concerns the acquisition of a specific type of image at a specific acquisition location in a specific manner, it will be understood that this routine or similar routines can be used to acquire video or other types of data (e.g., audio), whether in lieu of or in addition to such images. Furthermore, although the illustrated implementation acquires and uses information from the interior of the target building, it should be understood that other implementations can perform similar techniques on other types of data, including information on non-building structures and / or on the exterior of one or more target buildings of interest. Additionally, in some implementations, some routines can be executed on a mobile device used by a user to acquire image information, while other routines can be executed on one or more other computing devices (e.g., by a server computing system located remotely to such a mobile device and / or by one or more other computing devices at the location of the mobile device, such as using a distributed peer-to-peer approach with local interconnection at that location).

[0078] The illustrated implementation of the routine begins at block 705, where an instruction or information is received. At block 710, the routine determines whether the received instruction or information instructs the acquisition of data representing the interior of a building, and if not, proceeds to block 790. Otherwise, the routine proceeds to block 712 to receive an instruction to begin the image acquisition process at a first acquisition location (e.g., from a user of a mobile image acquisition device). After block 712, the routine proceeds to block 715 to acquire image information (e.g., one or more stereoscopic photographic images and / or panoramic images, such as using different acquisition orientations) at an acquisition location within the target building of interest. Routine 700 may also optionally capture additional information, such as annotations and / or information from the user regarding the captured images, IMU data, and / or other image acquisition metadata (e.g., regarding the movement of the computing device during image acquisition), the capture location, and / or the surrounding environment. Although not shown in the exemplary embodiments, in some embodiments, the routine may also determine and provide the user with one or more correction guidance prompts regarding image capture, such as corresponding to the motion of the mobile device, the quality of the captured sensor data and / or visual data, the relevant lighting / ambient conditions, the desirability of capturing one or more additional images from the acquisition location in different capture orientations, and any other suitable aspects of capturing one or more images.

[0079] After box 715 is completed, the routine continues to box 720 to determine if there are further acquisition locations, such as where images can be acquired based on relevant information provided by the user of the mobile device. If so, the routine continues to box 722 to optionally initiate the capture of linked information (such as acceleration data, additional visual data, etc.) as the mobile device moves along a path away from the current acquisition location toward the next acquisition location inside the building. As described elsewhere herein, the captured linked information may include additional sensor data (e.g., from one or more IMUs or inertial measurement units, on the mobile device or otherwise carried by the user, and / or additional image or video information) recorded during such movement. The initiation of the capture of such linked information may be performed in response to an explicit instruction from the user of the mobile device or based on one or more automated analyses of information recorded from the mobile device. Furthermore, in some implementations, during movement to the next acquisition location, the routine may optionally monitor the movement of the mobile device and determine and provide the user with one or more correction guidance prompts regarding the movement of the mobile device, the quality of the captured sensor data and / or video information, relevant lighting / ambient conditions, the desirability of capturing the next acquisition location, and any other appropriate aspects of capturing link information. Similarly, the routine may optionally obtain annotations and / or other information from the user regarding the travel path, such as information for later use in presenting information about the travel path or the resulting inter-panel connections. At box 724, the routine determines that the mobile device has reached the next acquisition location (e.g., based on instructions from the user, based on the user ceasing forward movement for at least a predetermined amount of time, etc.), uses it as the new current acquisition location, and returns to box 715 to perform acquisition location image acquisition activities for the new current acquisition location.

[0080] If it is determined in box 720 that no further acquisition locations exist for acquiring image information of the current building or other structure, the routine proceeds to box 781 to optionally analyze the acquisition location information of the building or other structure, such as to identify possible additional coverage (and / or other information) for acquiring the building interior. For example, the AIC system may provide the user with one or more notifications regarding information acquired during the capture of images for multiple acquisition locations, and optionally corresponding linked information, such as if it determines that one or more pieces or portions of the recorded information have insufficient or undesirable quality, or do not appear to provide complete coverage of the building. After box 781, the routine continues to box 783 to optionally preprocess the acquired images before they are subsequently used to generate related mapping information, so as to present the defined type of information using a specific format and / or in a specific manner (e.g., using a stereo linear planar format, etc.). After box 783, the routine continues to box 788 to store the images and any associated generated or acquired information for later use. Figures 4A-4B An example of a routine for generating mapping-related information about a building from such captured image information is shown.

[0081] If it is determined in box 710 that the instructions or other information stated in box 705 are not for acquiring images and other data representing the interior of a building, the routine continues to box 790 to perform any other instructed operations as appropriate, such as any household chores, to configure parameters to be used in various operations of the system (e.g., based at least in part on information specified by the user of the system, such as the user capturing one or more mobile devices inside the building, the operator user of the AIC system, etc.), to respond to requests for generated and stored information (e.g., to identify one or more captured images that match one or more specified search criteria, etc.), to obtain and store other information about the user of the system, etc.

[0082] After box 788 or 790, the routine proceeds to box 795 to determine whether to continue, such as until an explicit instruction to terminate is received, or only if an explicit instruction to continue is received. If it is determined to continue, the routine returns to box 705 to wait for further instructions or information; otherwise, it proceeds to step 799 and terminates.

[0083] The non-exclusive exemplary implementations described herein are further described in the following clauses.

[0084] A01. A computer-implemented method for performing automated operations on one or more computing systems, comprising:

[0085] The one or more computing systems acquire multiple images of various types taken at multiple acquisition locations associated with a building, wherein the multiple images include multiple stereoscopic photographs and multiple panoramic images, and wherein, for each of the multiple rooms in the building, the multiple acquisition locations include at least one acquisition location in the room.

[0086] Multiple image pairs are determined from the plurality of images by the one or more computing systems, wherein each image pair includes one of the plurality of stereoscopic photographs and one of the plurality of panoramic images, and the plurality of stereoscopic photographs and one of the plurality of panoramic images have overlapping visual coverage of at least one of the plurality of rooms;

[0087] Through the one or more computing systems and for each of the plurality of image pairs, by modifying the first image of the image pair to generate an enhanced image using data associated with the second image of the image pair, including selecting at least one type of attribute associated with the second image of the image pair, and adding data of the selected at least one type of attribute to the modified first image;

[0088] Using the one or more computing systems and at least one generated enhanced image, mapping information of the building is generated, at least in part, based on visual data from the plurality of images; and

[0089] At least some of the generated mapping information of the building is presented through the one or more computing systems.

[0090] A02. A computer-implemented method for performing automated operations on one or more computing systems, comprising:

[0091] The one or more computing systems acquire multiple images of various types taken at multiple acquisition locations associated with a house, wherein the multiple images of various types include multiple panoramic images in a rectangular format and multiple photographs in a stereo format, and wherein, for each of the multiple rooms of the house, the multiple acquisition locations include at least one acquisition location in the room.

[0092] The structural properties of one of the plurality of rooms are determined by the analysis of the first visual data in the panoramic images, at least in part, using the one or more computing systems and for each of the panoramic images, and the determined structural properties are also used to generate at least a portion of the room shapes of the plurality of rooms;

[0093] Using the one or more computing systems and for each of the stereoscopic photographs, at least a portion of the chromaticity attributes of one of the plurality of rooms are determined based, at least in part, on second visual data of the stereoscopic photographs;

[0094] Through the one or more computing systems, multiple image pairs are determined, wherein each image pair includes one of the multiple stereoscopic photographs and one of the multiple panoramic images, and one of the multiple stereoscopic photographs and one of the multiple panoramic images contributes visual features of one of the multiple rooms in the first visual data of the panoramic image and the second visual data of the stereoscopic photograph.

[0095] A first enhanced panoramic image is generated by modifying the panoramic image of the image pair to use the determined chromaticity attributes of the stereoscopic photograph of the image pair, and a second enhanced stereoscopic photograph is generated by modifying the stereoscopic photograph of the image pair to associate the determined structural attributes of the panoramic image of the image pair with the second enhanced stereoscopic photograph.

[0096] The mapping information of the house is generated, at least in part, based on visual data from the plurality of images, using the one or more computing systems, including generating a first enhanced panoramic image and a second enhanced stereoscopic photograph from at least some of the plurality of image pairs as part of generating at least a partial floor plan of the house; and

[0097] Presenting at least some of the generated mapping information of the house through the one or more computing systems includes presenting at least one generated first enhanced panoramic image and at least one generated second enhanced stereoscopic photograph.

[0098] A03. A computer-implemented method for performing automated operations on one or more computing systems, comprising:

[0099] Acquire multiple images of various types taken at multiple acquisition locations associated with a building, wherein, for each of multiple rooms in the building, the multiple acquisition locations include at least one acquisition location in the room;

[0100] Multiple image pairs are determined from the plurality of images, wherein each image pair includes a first image of a first type of the plurality of types and a second image of a different second type of the plurality of types, and wherein the first image and the second image in each image pair have overlapping visual coverage of at least one of the plurality of rooms;

[0101] For each of the plurality of image pairs, an enhanced image is generated by modifying the first image of the image pair to use data associated with the second image of the image pair, including selecting at least one type of attribute associated with the second image of the image pair, and adding data associated with the modified first image for the selected at least one type of attribute; and

[0102] At least some generated mapping information of the building is provided for display, wherein the at least some generated mapping information is based at least in part on at least one generated enhanced image.

[0103] A04. A computer-implemented method for performing automated operations on one or more computing systems, comprising:

[0104] Multiple images of various types are obtained by the one or more computing systems at multiple acquisition locations associated with a building, wherein the multiple images of various types include multiple stereoscopic photographs and multiple panoramic images, and wherein, for each of the multiple rooms in the building, the multiple acquisition locations include at least one acquisition location in the room;

[0105] Through the one or more computing systems, and for each of a plurality of image pairs, each image pair including one of the plurality of stereoscopic photographs and one of the plurality of panoramic images, the one of the plurality of stereoscopic photographs and the one of the plurality of panoramic images having overlapping visual overlay of at least one of the plurality of rooms, an enhanced image is generated by exchanging attribute data between the stereoscopic photograph and the panoramic image of the image pair, by:

[0106] An enhanced stereoscopic photograph is generated by modifying one of the stereoscopic photographs in the image pair using data from a first type of attribute of the panoramic image in the image pair, using the one or more computing systems mentioned above; and

[0107] An enhanced panoramic image is generated by modifying one of the panoramic images in the image pair using data from a second type of attribute of the stereoscopic photograph of the image pair, wherein the first type of attribute and the second type of attribute are different, through the one or more computing systems.

[0108] The one or more computing systems provide at least some generated mapping information of the building for display, wherein the at least some generated mapping information is based at least in part on at least some of the generated enhanced images.

[0109] A05. A computer-implemented method as described in any one of clauses A01-A04, wherein each of the panoramic images includes 360-degree visual coverage around a vertical axis, and wherein determining the plurality of image pairs further includes:

[0110] Generating multiple sub-images from the plurality of panoramic images using the one or more computing systems includes: for each of the panoramic images, generating multiple sub-images, each of the multiple sub-images including a different subset of the visual data of the panoramic image and having a stereo format;

[0111] Through the one or more computing systems and for each of the plurality of sub-images, generate one or more first global features that describe the visual data of the sub-image as a whole and a plurality of first local features that describe the various parts of the visual data of the sub-image;

[0112] Through the one or more computing systems and for each of the plurality of photographs, one or more second global features are generated to describe the visual data of the photograph as a whole and a plurality of second local features are generated to describe the various parts of the visual data of the photograph;

[0113] Through the one or more computing systems and for each of the plurality of photos, a group of the plurality of sub-images having first global features that match the second global features of the photo is determined, and a sub-image is selected from the determined group, wherein the plurality of first local features of the determined group best match the plurality of second local features of the photo.

[0114] Through the one or more computing systems and for each of the plurality of photographs, generate one of the image pairs including the photograph and including the panoramic image, and generate a selected sub-image of the photograph from the panoramic image; and

[0115] The first visual data of a panoramic image of the image pair and the second visual data of a stereoscopic photograph of the image pair are aligned using the one or more computing systems and for each of the plurality of image pairs to share a common coordinate system.

[0116] Furthermore, the generation of the first enhanced panoramic image and the second enhanced stereoscopic photograph of each of the plurality of image pairs includes using first visual data of the one panoramic image of the aligned image pair and second visual data of the one stereoscopic photograph of the image pair.

[0117] A06. The computer-implemented method as described in Clause A05 further includes:

[0118] Using the one or more computing systems and for each of at least some of the plurality of image pairs, analyze a subset of the panoramic image of the image pair visible in the stereoscopic photograph of the image pair to determine characteristics associated with the subset of the panoramic image, and combine the characteristics determined from the at least some image pairs to identify image characteristics associated with the area of ​​interest of the house.

[0119] An additional image of interest (OPI) is generated by using the one or more computing systems and, for each of the one or more additional panoramic images that are not part of the at least some image pairs, by selecting a subset of the additional panoramic images to include in the additional OPI using the identified image features; and

[0120] The one or more computing systems present at least one generated additional image of interest.

[0121] A07. A computer-implemented method as described in any one of clauses A01-A06, wherein the first image of one image pair in the image pair is the panoramic image of the image pair, and the second image of the image pair is the stereoscopic photograph of the image pair, wherein the method further comprises determining the chromaticity attribute of the stereoscopic photograph of the image pair, and wherein generating the enhanced image of the image pair comprises generating the enhanced panoramic image by modifying the panoramic image of the image pair to use the determined chromaticity attribute instead of other chromaticity attributes of the panoramic image of the image pair.

[0122] A08. The computer-implemented method as described in Clause A07 further includes:

[0123] Using the one or more computing systems, at least one of the visible structural shapes in the panoramic image of the image pair and the visible structural wall features in the panoramic image of the image pair is determined;

[0124] Using the one or more computing systems, determine one or more locations in the panoramic image of at least one of the structural shapes or structural wall features; and

[0125] An enhanced stereoscopic photograph of the image pair is generated by modifying the stereoscopic photograph of the image pair to associate at least one of the structural shape or structural wall features at one or more other locations in the visual data of the modified stereoscopic photograph, using the one or more computing systems.

[0126] Furthermore, the generation of building mapping information also utilizes the generated enhanced panoramic images and the generated enhanced stereoscopic photographs.

[0127] A09. A computer-implemented method as described in any one of clauses A01-A08, wherein the first image of one image pair in the image pair is the stereoscopic photograph of the image pair, and the second image in the image pair is the panoramic image of the image pair, wherein the method further comprises determining at least one of a visible structural shape and a visible structural wall feature in the panoramic image of the image pair, and determining one or more locations in the panoramic image of the at least one structural shape or structural wall feature, and wherein generating the enhanced image of the image pair comprises generating the enhanced stereoscopic photograph by modifying the stereoscopic photograph of the image pair to associate the at least one structural shape or structural wall feature at one or more other locations in the visual data of the modified stereoscopic photograph of the at least one structural shape or structural wall feature.

[0128] A10. A computer-implemented method as described in any one of clauses A01-A09, wherein the first image of one image pair in the image pair is the panoramic image of the image pair, and the second image of the image pair is the stereoscopic photograph of the image pair, wherein the method further comprises:

[0129] Using the one or more computing systems and for each of at least some of the plurality of image pairs other than the one image pair, analyze a subset of the panoramic images of the image pair visible in the stereoscopic photographs of the image pairs to determine characteristics associated with the subset of the panoramic images; and

[0130] Using the one or more computing systems, features determined from the at least some image pairs are combined to identify image features associated with some regions.

[0131] Furthermore, generating the enhanced image of the image pair includes selecting a subset of the panoramic images of the image pair for use as the enhanced image of the image pair using the recognized image features and the visual data of the stereoscopic photographs of the image pair.

[0132] A11. A computer-implemented method as described in any one of clauses A01-A10, wherein each of the panoramic images includes 360-degree visual coverage around a vertical axis, and wherein determining the plurality of image pairs further includes:

[0133] Generating multiple sub-images from the plurality of panoramic images using the one or more computing systems includes: for each of the panoramic images, generating multiple sub-images, each of the multiple sub-images including a different subset of the visual data of the panoramic image and being in a stereo format;

[0134] Through the one or more computing systems and for each of the plurality of sub-images, generate one or more first global features that describe the visual data of the sub-image as a whole, and a plurality of first local features that describe the various parts of the visual data of the sub-image;

[0135] Through the one or more computing systems and for each of the plurality of stereoscopic images, generate one or more second global features that describe the visual data of the stereoscopic image as a whole, and a plurality of second local features that describe the various parts of the visual data of the stereoscopic image;

[0136] Using the one or more computing systems and for each of at least some of the plurality of stereoscopic images, a group of the plurality of sub-images having first global features that match the second global features of the stereoscopic image is determined, and a sub-image is selected from the determined group, wherein the plurality of first local features of the determined group match the plurality of second local features of the stereoscopic image; and

[0137] Through one or more computing systems and for each of the at least some stereoscopic photographs, generate one of the image pairs including the stereoscopic photograph and the panoramic image, and generate a selected sub-image of the photograph from the panoramic image.

[0138] A12. A computer-implemented method as described in any one of clauses A01-A11, wherein at least some of the generated mapping information provided is an enhanced image of the at least one generated image, and wherein presenting the at least some of the generated mapping information by the one or more computing systems includes sending the at least some of the generated mapping information to a client computing device via the one or more computing systems and via one or more computer networks for display to one or more users on the client computing device.

[0139] A13. A computer-implemented method as described in any one of clauses A01-A12, wherein each of the plurality of stereoscopic photographs is in a stereoscopic format and has a viewing angle of less than 90 degrees, wherein each of the plurality of panoramic images is in an equal rectangular format and has a viewing angle of at least 180 degrees, wherein the plurality of acquisition locations include one or more acquisition locations outside the building, wherein determining the plurality of image pairs includes analyzing visual data of the plurality of images to identify matching visual features in the stereoscopic photograph and the panoramic image of each of the plurality of image pairs, and wherein the generated mapping information includes at least partial floor plans of the building based on the visual data of the plurality of images.

[0140] A14. A computer-implemented method as described in any one of clauses A01-A13, wherein the plurality of images of the first type comprise photographs in a stereoscopic format and having a viewing angle of less than 90 degrees, wherein the plurality of images of the second type comprise panoramic images in an equal rectangular format and having a viewing angle of at least 180 degrees, and wherein the method further comprises generating the mapping information of the building based at least in part on visual data of the plurality of images and includes using at least one generated enhanced image.

[0141] A15. The computer-implemented method as described in any one of clauses A01-A14, wherein at least some of the generated mapping information provided is an enhanced image of said at least one generated image.

[0142] A16. A computer-implemented method as described in any one of clauses A01-A15, wherein the first image of one of the image pairs is a panoramic image, and the second image of the image pair is a stereoscopic photograph, wherein the automatic operation further includes determining the chromaticity attribute of the stereoscopic photograph of the image pair, and wherein generating the enhanced image of the image pair comprises generating the enhanced panoramic image by modifying the panoramic image of the image pair to use the determined chromaticity attribute.

[0143] A17. The computer-implemented method as described in clause A16, wherein modifying the panoramic image of the image pair to use the determined chromaticity attribute further includes preserving the luminance attribute of the panoramic image of the image pair in the modified panoramic image.

[0144] A18. A computer-implemented method as described in any one of clauses A01-A17, wherein the first image of one of the image pairs is a stereoscopic photograph, and the second image of the image pair is a panoramic image, wherein the automatic operation further includes determining one or more structural shapes visible in the panoramic image and one or more locations of the one or more structural shapes in the panoramic image, and wherein generating the enhanced image of the image pair comprises generating the enhanced stereoscopic photograph by modifying the stereoscopic photograph to associate at least one of the one or more structural shapes at one or more other locations in the visual data of the modified stereoscopic photograph of the at least one structural shape.

[0145] A19. A computer-implemented method as described in any one of clauses A01-A18, wherein the first image of one of the image pairs is a stereoscopic photograph, and the second image of the image pair is a panoramic image, wherein the automatic operation further includes determining one or more structural wall features visible in the panoramic image of the image pair and one or more locations of the one or more structural wall features in the panoramic image, and wherein generating the enhanced image of the image pair comprises generating an enhanced stereoscopic photograph by modifying the stereoscopic photograph to associate at least one of the one or more structural wall features at one or more other locations in the visual data of the modified stereoscopic photograph of the at least one structural wall feature.

[0146] A20. The computer-implemented method as described in any one of clauses A01-A19

[0147] Wherein, the first image in one of the image pairs is a panoramic image, and the second image in the image pair is a stereoscopic photograph, wherein the automatic operation further includes:

[0148] For each of at least some of the plurality of image pairs other than the one image pair, a subset of the panoramic images of the image pair visible in the stereoscopic photographs of the image pair is analyzed to determine characteristics associated with the subset of the panoramic images; and

[0149] The features determined from the at least some image pairs are combined to identify image features associated with some areas of the building.

[0150] Furthermore, generating the enhanced image of the image pair includes selecting a subset of the panoramic images of the image pair for use as the enhanced image of the image pair using the recognized image features and the visual data of the stereoscopic photographs of the image pair.

[0151] A21. The computer-implemented method as described in clause A20, wherein the determined characteristics are associated with a subset of the buildings, at least one of the subsets of buildings having visual data of one or more defined regions of the buildings, or having visual data with details of one or more defined subject areas of interest, or having visual data matching characteristics of one or more defined types.

[0152] A22. A computer-implemented method as described in any of clauses A01-A21, wherein generating the enhanced image of one of the image pairs further comprises selecting one or more attributes from a group comprising at least light balance and saturation, and sharpness and style, and adding data of each of the selected one or more attributes from the second image of the image pair to the modified first image.

[0153] A23. A computer-implemented method as described in any one of clauses A01-A22, wherein generating the enhanced image of one of the image pairs further comprises selecting one or more attributes from a group comprising at least user annotations and semantic tags, and adding data of each of the selected one or more attributes from the second image in the image pair associated with the modified first image.

[0154] A24. A computer-implemented method as described in any one of clauses A01-A23, wherein the plurality of image types include at least one of a first type and a second type, wherein the first type and the second type include daytime and nighttime versions of at least some public areas associated with the building, or the first type and the second type include visual and non-visual data of at least some public areas associated with the building, or the first type and the second type include versions acquired at different times of at least some public areas associated with the building, and wherein generating the enhanced image of at least one of the plurality of image pairs includes exchanging data between at least one of the daytime version and the nighttime version, or the visual data and the non-visual data, or the versions acquired at different times.

[0155] A25. A computer-implemented method as described in any one of clauses A01-A24, wherein generating the enhanced image of one of the image pairs further includes adding data of one or more depth attributes from the second image in the image pair associated with the modified first image.

[0156] A26. A computer-implemented method as described in any one of clauses A01-A25, wherein generating the enhanced image of one of the image pairs further includes adding data of one or more positional attributes from the second image in the image pair associated with the modified first image.

[0157] A27. A computer-implemented method as described in any one of clauses A01-A26, wherein determining one of the image pairs further includes using data of one or more depth attributes from at least one of the first image or the second image in the image pair.

[0158] A28. A computer-implemented method as described in any one of clauses A01-A27, wherein determining one of the image pairs further includes using data of one or more positional attributes from at least one of the first image or the second image in the image pair.

[0159] A29. The computer-implemented method as described in any one of clauses A01-A28

[0160] Among these, one of the multiple types is a panoramic image in a rectangular format, and another of the multiple types is a stereoscopic photograph in a stereoscopic format, and determining the multiple image pairs further includes:

[0161] Generating multiple sub-images from multiple panoramic images from the plurality of images includes: for each of the panoramic images, generating multiple sub-images, each of the multiple sub-images including a different subset of the visual data of the panoramic image and being in a stereo format;

[0162] For each of the plurality of sub-images, generate one or more first global features that describe the visual data of the sub-image as a whole, and a plurality of first local features that describe the various parts of the visual data of the sub-image;

[0163] For each of the plurality of stereoscopic photographs of the plurality of images, generate one or more second global features that describe the visual data of the stereoscopic photograph as a whole, and a plurality of second local features that describe the various parts of the visual data of the stereoscopic photograph;

[0164] For each of at least some of the plurality of stereoscopic images, a group of the plurality of sub-images having first global features that match the second global features of the stereoscopic image is determined, and a sub-image is selected from the determined group, wherein the plurality of first local features of the determined group match the plurality of second local features of the stereoscopic image; and

[0165] For each of the at least some stereoscopic photographs, generate one image pair including the stereoscopic photograph and one image pair including the panoramic image, and generate a selected sub-image of the photograph from the panoramic image.

[0166] A30. The computer-implemented method as described in any one of clauses A01-A29

[0167] The automatic operation further includes at least one of the following:

[0168] By identifying one or more additional second images of the second type, each of which has visual overlap with the first image in the image pair, and by combining data from the second image in the image pair and the one or more additional second images of at least one selected type of attribute, the enhancement image of one of the plurality of image pairs is generated, wherein the addition of data associated with the modified first image of the image pair uses the combined data; or

[0169] The enhanced image of one image pair in the plurality of image pairs is generated by using the noise distribution of the second image in the image pair to enhance one or more portions of the modified first image in the image pair; or

[0170] For one of the plurality of image pairs, the generation of at least some mapping information is performed by associating the second image in the image pair with one or more locations on the modified first image of the image pair as user-selectable points of interest; or

[0171] At least a partial floor plan of the building is generated using visual data from the plurality of images, and for one of the plurality of image pairs, the generation of at least some mapping information is performed by associating at least one of the first image or the second image in the image pair with one or more locations on the at least partial floor plan as user-selectable points of interest; or

[0172] After providing the at least some of the generated mapping information, the generated mapping information is updated based on one or more additional acquired images at one or more additional acquisition locations associated with the building, including generating one or more additional enhanced images by exchanging attribute data between images of one or more additional image pairs, each of the additional image pairs including at least one of the additional acquired images; or

[0173] For one of the plurality of image pairs, the generation of the at least some mapping information is performed by generating an additional enhanced image having visual data from a combination of the first image from the image pair and the second image from the image pair; or

[0174] For one of the plurality of image pairs, at least some mapping information is generated by using one or more machine learning models and visual data from the first image and the second image in the image pair to provide an increase in the apparent resolution of the additional enhanced image.

[0175] A31. The computer-implemented method as described in any one of clauses A01-A30

[0176] It also includes a user's client computing device, wherein the plurality of acquisition locations include one or more acquisition locations outside the building, wherein the automatic operation includes generating the mapping information of the building by using visual data from the plurality of images to generate at least a partial floor plan of the building, wherein providing the at least some of the generated mapping information of the building includes sending the at least some of the generated mapping information of the building to the client computing device via one or more computer networks, and wherein the automatic operation further includes: receiving and displaying the provided at least some of the generated mapping information on the client computing device; and sending information from the interaction between the user and user-selectable controls on the client computing device to the one or more computing systems via the client computing device, resulting in modification of the information of the building displayed on the client computing device.

[0177] A32. A computer-implemented method as described in any one of clauses A01-A31, wherein the method further comprises generating the mapping information of the building based at least in part on visual data of the plurality of images, and includes using the at least some of the generated enhanced images, wherein the at least some of the generated enhanced images include one or more generated enhanced stereoscopic photographs and one or more generated enhanced panoramic images.

[0178] A33. The computer-implemented method as described in any one of clauses A01-A32, wherein at least some of the generated mapping information provided is one or more of the at least some of the generated enhanced images.

[0179] A34. A computer-implemented method as described in any one of clauses A01-A33, wherein the plurality of images of the first type include the plurality of stereoscopic photographs, each of the plurality of stereoscopic photographs being in a stereoscopic format and having a viewing angle of less than 90 degrees, wherein the plurality of images of the second type include the plurality of panoramic images, each of the plurality of panoramic images being in an equal rectangular format and having a viewing angle of at least 180 degrees, and wherein the automatic operation further includes determining the plurality of image pairs by the one or more computing systems, including analyzing visual data of the plurality of images to identify matching visual features in the stereoscopic photograph and the panoramic image of each of the plurality of image pairs.

[0180] A35. A computer-implemented method as described in any one of clauses A01-A34, wherein the plurality of acquisition locations include one or more acquisition locations outside the building, wherein the automatic operation includes generating the mapping information of the building by generating at least a partial floor plan of the building using visual data from the plurality of images, wherein providing the at least some of the generated mapping information of the building includes sending the at least some of the generated mapping information of the building to a user's client computing device via one or more computer networks, and wherein the automatic operation further includes: receiving and displaying the provided at least some of the generated mapping information on the client computing device; and sending information from the interaction between the user and a user-selectable control on the client computing device to the one or more computing systems via the client computing device, resulting in modification of the information displayed on the client computing device of the building.

[0181] A36. A computer-implemented method comprising multiple steps of performing automated operations, said automated operations implementing techniques substantially as disclosed herein.

[0182] B01. A non-transitory computer-readable medium having stored executable software instructions and / or other stored content, the executable software instructions and / or other stored content causing one or more computing systems to perform automatic operation of the methods implementing any one of the provisions A01-A36.

[0183] B02. A non-transitory computer-readable medium having stored executable software instructions and / or other stored content, the stored executable software instructions and / or other stored content causing one or more computing systems to perform automated operations implementing techniques substantially as disclosed herein.

[0184] C01. One or more computing systems, including one or more hardware processors and one or more memories having stored instructions, which, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automatic operations of any one of the methods implementing clauses A01-A36.

[0185] CO2. One or more computing systems, including one or more hardware processors and one or more memories having stored instructions, which, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations implementing techniques substantially as disclosed herein.

[0186] D01. A computer program adapted to perform any one of the methods of clauses A01-A36 when the computer program is run on a computer.

[0187] Aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems / devices), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations thereof, can be implemented by computer-readable program instructions. It will be further understood that in some implementations, the functionality provided by the above-described routines may be provided in alternative ways, such as splitting among more routines or merging into fewer routines. Similarly, in some implementations, the described routines may provide more or less functionality than described, such as when other described routines are lacking or include such functionality, or when the amount of functionality provided changes. Furthermore, while various operations may be shown as being performed in a particular manner (e.g., serial or parallel, or synchronous or asynchronous) and / or in a particular order, in other implementations, operations may be performed in other orders and other manners. Any data structures discussed above may also be constructed in different ways, such as by dividing a single data structure into multiple data structures and / or by merging multiple data structures into a single data structure. Similarly, in some implementations, the data structure shown may store more or less information than described, for example, when other data structures shown are missing or include such information, or when the amount or type of information stored changes.

[0188] As can be understood from the foregoing, although specific embodiments have been described herein for illustrative purposes, various modifications may be made without departing from the spirit and scope of the invention. Therefore, the invention is not limited except by the corresponding claims and the elements referenced by those claims. Furthermore, while certain aspects of the invention may be presented at certain times in the form of certain claims, the inventors have considered various aspects of the invention to be in the form of any available claims. For example, while only some aspects of the invention may be described as embodied in a computer-readable medium at a particular time, other aspects may also be embodied in the same way.

Claims

1. A system comprising: one or more hardware processors of one or more computing systems; and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations comprising at least: acquiring a plurality of images of a plurality of types taken at a plurality of capture locations associated with a building, wherein, for each of a plurality of rooms of the building, the plurality of capture locations includes at least one capture location in the room; determining, from the plurality of images, a plurality of image pairs, wherein each image pair includes a first image of a first type of the plurality of types and a second image of a second, different type of the plurality of types, and wherein the first and second images of each image pair have overlapping visual coverage of at least one room of the plurality of rooms, wherein the first type of the plurality of images includes photos in a stereoscopic format and having a viewing angle of less than 90 degrees, wherein the second type of the plurality of images includes panoramic images in an equirectangular format and having a viewing angle of at least 180 degrees; for each of the plurality of image pairs, generating an augmented image by modifying the first image of the image pair to use data associated with the second image of the image pair, including selecting at least one type of attribute associated with the second image of the image pair, and adding data associated with the modified first image for the selected at least one type of attribute; generating, based at least in part on at least one generated augmented image and based at least in part on visual data of the plurality of images, mapping information of the building for display; and providing at least some generated mapping information of the building for display, wherein the provided at least some generated mapping information includes the at least one generated augmented image.

2. The system of claim 1, wherein, the first image of one of the image pairs is one of the panoramic images in the equirectangular format and the second image of the one of the image pairs is one of the photos in the stereoscopic format, wherein the automated operations further comprise determining a chrominance attribute of the one photo of the one of the image pairs, and wherein generating the augmented image of the one of the image pairs includes generating an augmented panoramic image by modifying the one panoramic image of the one of the image pairs to use the determined chrominance attribute.

3. The system of claim 2, wherein, modifying the one panoramic image of the one of the image pairs to use the determined chrominance attribute further comprises preserving a luminance attribute of the one panoramic image of the one of the image pairs in the modified panoramic image.

4. The system of claim 1, wherein, The first image of one of the image pairs is one of the photographs in the stereoscopic format and the second image in the one of the image pairs is one of the equirectangular format panorama images, wherein the automated operations further comprise determining one or more structural shape visible in the one panorama image and one or more locations of the one or more structural shape in the one panorama image, and wherein generating the augmented image of the one of the image pairs comprises generating an augmented photograph in the stereoscopic format by modifying the one photograph to associate at least one of the one or more structural shape at one or more other locations in the modified photograph's visual data.

5. The system of claim 1, wherein, The first image of one of the image pairs is one of the photographs in the stereoscopic format and the second image in the one of the image pairs is one of the equirectangular format panorama images, wherein the automated operations further comprise determining one or more structural wall features visible in the one panorama image of the one of the image pairs and one or more locations of the one or more structural wall features in the one panorama image, and wherein generating the augmented image of the one of the image pairs comprises generating an augmented photograph in the stereoscopic format by modifying the one photograph to associate at least one of the one or more structural wall features at one or more other locations in the modified photograph's visual data of the at least one structural wall feature of the one or more structural wall features.

6. The system of claim 1, wherein, The first image of each of the image pairs is one of the equirectangular format panorama images and the second image in each of the image pairs is one of the photographs in the stereoscopic format, wherein the automated operations further comprise: For each of some of the plurality of image pairs, analyzing a subset of the one panorama image of the image pair visible in the one photograph of the image pair to determine a characteristic associated with the subset of the one panorama image; and combining the characteristics determined from some of the plurality of image pairs to identify image characteristics associated with some regions of the building, and wherein generating the augmented image of one of the image pairs comprises using the identified image characteristics and visual data of the one photograph of the one of the image pairs to select a subset of the one panorama image of the one of the image pairs to use as the augmented image of the one of the image pairs. The first image of one of the image pairs is one of the photographs in the stereoscopic format and the second image in the one of the image pairs is one of the equirectangular format panorama images, wherein the automated operations further comprise determining one or more structural shape visible in the one panorama image and one or more locations of the one or more structural shape in the one panorama image, and wherein generating the augmented image of the one of the image pairs comprises generating an augmented photograph in the stereoscopic format by modifying the one photograph to associate at least one of the one or more structural shape at one or more other locations in the modified photograph's visual data. The first image of one of the image pairs is one of the photographs in the stereoscopic format and the second image in the one of the image pairs is one of the equirectangular format panorama images, wherein the automated operations further comprise determining one or more structural wall features visible in the one panorama image of the one of the image pairs and one or more locations of the one or more structural wall features in the one panorama image, and wherein generating the augmented image of the one of the image pairs comprises generating an augmented photograph in the stereoscopic format by modifying the one photograph to associate at least one of the one or more structural wall features at one or more other locations in the modified photograph's visual data of the at least one structural wall feature of the one or more structural wall features. The first image of each of the image pairs is one of the equirectangular format panorama images and the second image in each of the image pairs is one of the photographs in the stereoscopic format, wherein the automated operations further comprise: For each of some of the plurality of image pairs, analyzing a subset of the one panorama image of the image pair visible in the one photograph of the image pair to determine a characteristic associated with the subset of the one panorama image; and combining the characteristics determined from some of the plurality of image pairs to identify image characteristics associated with some regions of the building, and wherein generating the augmented image of one of the image pairs comprises using the identified image characteristics and visual data of the one photograph of the one of the image pairs to select a subset of the one panorama image of the one of the image pairs to use as the augmented image of the one of the image pairs.

7. The system of claim 1, wherein, Generating the augmented image of one of the pairs of images further comprises selecting one or more attributes from a group of attributes comprising at least light balance and saturation and sharpness and style, and adding data of each of the selected one or more attributes from the second image of the one of the pairs of images to the modified first image of the one of the pairs of images.

8. The system of claim 1, wherein, Generating the augmented image of one of the pairs of images further comprises selecting one or more attributes from a group of attributes comprising at least user annotations and semantic labels, and adding data of each of the selected one or more attributes from the second image of the one of the pairs of images associated with the modified first image of the one of the pairs of images.

9. The system of claim 1, wherein, The multiple types of images comprise at least one of a first type and a second type, wherein the first type and the second type comprise versions taken during daytime and nighttime of at least some common areas associated with the building, or the first type and the second type comprise visual data and non-visual data of at least some common areas associated with the building, or the first type and the second type comprise versions taken at different times of at least some common areas associated with the building, and wherein generating the augmented image of at least one of the multiple pairs of images comprises exchanging data between at least one of: versions taken during the daytime and the nighttime, or the visual data and the non-visual data, or versions taken at different times.

10. The system of claim 1, wherein, Generating the augmented image of one of the pairs of images further comprises at least one of: adding data of one or more depth attributes from the second image of the one of the pairs of images associated with the modified first image, or adding data of one or more location attributes from the second image of the one of the pairs of images associated with the modified first image, and wherein determining the one of the pairs of images further comprises at least one of: using data of one or more depth attributes of at least one of the first image or the second image from the one of the pairs of images, or using data of one or more location attributes of at least one of the first image or the second image from the one of the pairs of images.

11. The system of claim 1, wherein, Determining the multiple pairs of images further comprises: generating a plurality of sub-images from the equirectangular format panoramic image, comprising: for each of the panoramic images, generating a plurality of sub-images, each of the plurality of sub-images comprising a different subset of the visual data of the panoramic image and being a stereographic format; for each of the plurality of sub-images, generating one or more first global features describing the visual data of the sub-image as a whole, and a plurality of first local features describing individual portions of the visual data of the sub-image; for each of the plurality of sub-images, generating one or more first global features describing the visual data of the sub-image as a whole, and a plurality of first local features describing individual portions of the visual data of the sub-image; for each of the photos in the stereoscopic format, generating one or more second global features that describe the visual data of the photo as a whole, and a plurality of second local features that describe individual portions of the visual data of the photo; for each of at least some of the photos in the stereoscopic format, determining a group of the plurality of sub-images having first global features that match the second global features of the photo, and selecting one sub-image from the determined group, the plurality of first local features of the determined group matching the plurality of second local features of the photo; and for each of at least some of the photos in the stereoscopic format, generating one of the image pairs including the photo and the panoramic image, the selected one sub-image of the photo being generated from the panoramic image.

12. The system of claim 1, wherein, The automated operations further include at least one of: performing the generating of the enhanced image of one of the plurality of image pairs by identifying one or more additional second images of the second type, each of the second images having overlapping visual coverage with the first image of the one image pair, and by combining data from the second image of the one image pair and the one or more additional second images of the selected at least one type of attribute, and wherein the addition of the data associated with the modified first image of the one image pair uses the combined data; or performing the generating of the enhanced image of one of the plurality of image pairs by using a noise distribution of the second image of the one image pair to accentuate one or more portions of the modified first image of the one image pair; or for one of the plurality of image pairs, performing the generating of the survey information by associating the second image of the one image pair with one or more locations on the modified first image of the one image pair as user-selectable points of interest; or performing the generating of the survey information by using visual data of the plurality of images to generate at least a partial floor plan of the building, and for one of the plurality of image pairs, by associating at least one of the first image of the one of the image pair or the second image of the one of the image pair with one or more locations on the at least partial floor plan as user-selectable points of interest; or after providing at least some of the generated survey information, updating the generated survey information based on one or more additional captured images at one or more additional capture locations associated with the building, including by generating one or more additional enhanced images by exchanging attribute data between images of one or more additional image pairs, each of the additional image pairs including at least one of the additional captured images; or generating the mapping information for one of the pairs of images by generating an additional augmented image having visual data combined from the first image in the one of the pairs of images and the second image in the one of the pairs of images; or generating the mapping information for one of the pairs of images by generating an additional augmented image using one or more machine learning models and visual data from the first image in the one of the pairs of images and the second image in the one of the pairs of images to provide an increase in apparent resolution of the additional augmented image.

13. The system of claim 1, further comprising a client computing device of a user, wherein, the plurality of capture locations includes one or more capture locations outside of the building, wherein generating the mapping information includes generating at least a partial floor plan of the building using visual data of the plurality of images, wherein providing at least some of the generated mapping information for the building includes transmitting the at least some of the generated mapping information for the building to the client computing device over one or more computer networks, and wherein the automated operations further include receiving and displaying at least some of the provided generated mapping information on the client computing device by the client computing device, and transmitting information from interactions of the user with user-selectable controls on the client computing device to the one or more computing systems to cause modification of information displayed on the client computing device for the building.

14. A computer-implemented method comprising: obtaining, by one or more computing systems, a plurality of images of a plurality of types taken at a plurality of capture locations associated with a building, wherein the plurality of images includes a plurality of photos in a stereoscopic format and having a viewing angle of less than 90 degrees, and a plurality of panoramic images in an equirectangular format and having a viewing angle of at least 180 degrees, and wherein, for each of a plurality of rooms of the building, the plurality of capture locations includes at least one capture location in the room; determining, by the one or more computing systems and from the plurality of images, a plurality of pairs of images, wherein each pair of images includes one of the plurality of photos in the stereoscopic format and one of the plurality of panoramic images in the equirectangular format, the one of the plurality of photos in the stereoscopic format and the one of the plurality of panoramic images in the equirectangular format having overlapping visual coverage of at least one of the plurality of rooms; generating, by the one or more computing systems and for each of the pairs of images, an augmented image by modifying a first image of the pair of images to use data associated with a second image of the pair of images, including selecting at least one type of attribute associated with the second image of the pair of images, and adding data of the selected at least one type of attribute to the modified first image; generating, by the one or more computing systems, mapping information for the building based at least in part on the at least one generated augmented image and visual data of the plurality of images; and presenting, by the one or more computing systems, at least some of the generated mapping information for the building, wherein at least some of the presented generated mapping information includes the at least one generated augmented image.

15. A non-transitory computer-readable medium having stored contents that cause one or more computing systems to perform automated operations comprising at least: obtaining, by the one or more computing systems, a plurality of images of a plurality of types taken at a plurality of capture locations associated with a building, wherein the plurality of images of the plurality of types include a plurality of photos in a stereoscopic format and having a viewing angle of less than 90 degrees, and include a plurality of panoramic images in an equirectangular format and having a viewing angle of at least 180 degrees, and wherein, for each of a plurality of rooms of the building, the plurality of capture locations includes at least one capture location in the room; generating, by the one or more computing systems, for each image pair of a plurality of image pairs, each image pair including one photo of the plurality of photos in the stereoscopic format and one panoramic image of the plurality of panoramic images in the equirectangular format, the one photo of the plurality of photos in the stereoscopic format and the one panoramic image of the plurality of panoramic images in the equirectangular format having overlapping visual coverage of at least one of the plurality of rooms, an augmented image by exchanging attribute data between the one photo of the plurality of photos and the one panoramic image of the plurality of panoramic images of the image pair by: generating, by the one or more computing systems, an augmented photo in the stereoscopic format by modifying the one photo of the plurality of photos in the image pair to use data of a first type of attribute from the one panoramic image of the plurality of panoramic images in the image pair; and generating, by the one or more computing systems, an augmented panoramic image in the equirectangular format by modifying the one panoramic image of the plurality of panoramic images in the image pair to use data of a second type of attribute from the one photo of the plurality of photos of the image pair, wherein the first type of attribute and the second type of attribute are different; and providing, by the one or more computing systems, at least some generated mapping information for the building for display, wherein at least some of the generated mapping information is based at least in part on at least some of the generated augmented images.

Images