Automatically determining building dimensions using acquired building image analysis

Abstract

Techniques are described for performing automatic operations using a computing device to analyze visual data of images acquired at a building to determine building information including building dimensions. Automatic determination of building size and other building information may include determining an estimated camera height for one or more camera devices based on an identified visible structure building object of a defined type while acquiring an image; further determining the resulting building size using the determined image scale information; and associating the building size data with a plan view generated from the image analysis. Information about such determined buildings may be used in a variety of automated ways, including for controlling device navigation (e.g., autonomous vehicles), for display on client devices in corresponding graphical user interfaces, for further analysis to identify shared and / or aggregated features, and the like.

Description

Technical Field

[0001] The following disclosure generally relates to techniques for automatically analyzing visual data of an image acquired at a building to determine building information, including building dimensions, and subsequently using such information in one or more automated ways to determine scale information of a building image using a defined type of visible structured building object, to use the image scale information to determine the final building dimensions of walls and other elements visible in the image (e.g., for use with a floor plan generated from the image analysis), and to provide navigation data for the building based on the determined building information and by automatically generating a description of the similarity. Figure 1 and starting from). Background Art

[0002] In various situations, such as building analysis, property inspection, real estate acquisition and development, general contracting, improvement cost estimation, etc., it may be desirable to know the interior of a house or other building without physically traveling to and entering the building. However, it may be difficult to effectively capture, represent, and use such building interior information, including identifying buildings that meet the criteria of interest and displaying visual information captured inside the building to a user at a remote location (e.g., enabling the user to understand the layout and other details of the interior, including controlling the display in a manner selected by the user). Moreover, even when the user is present on the building, it is difficult to effectively navigate the building and determine information about the building, which is not easily understood. Although a floor plan of a building can provide some information about the layout and other details of the building interior, using such a floor plan has several drawbacks, including difficulty in construction and maintenance, difficulty in accurately scaling and filling in information about the interior of rooms, difficulty in visualization and other uses, etc. Brief Description of the Drawings

[0003] Figure 1 Includes diagrams depicting an exemplary building interior environment and one or more computing systems used in embodiments of the present disclosure, including generating and presenting information representing the building interior based in part on the determined building dimension data.

[0004] Figures 2A to 2L Illustrates an example of automatically analyzing visual data of an image acquired at a building to determine building information including building dimensions and subsequently using such information in one or more automated ways.

[0005] Figure 3 Is a block diagram of a computing system showing an embodiment of a system suitable for implementing at least some of the techniques described in the present disclosure.

[0006] Figure 4Shows an exemplary embodiment of a flowchart of an Image Capture and Analysis (ICA) system routine according to an embodiment of the present disclosure.

[0007] Figures 5A to 5B Shows an exemplary embodiment of a flowchart of a Mapping Information Generation Manager (MIGM) system routine according to an embodiment of the present disclosure.

[0008] Figure 6 Shows an exemplary embodiment of a flowchart of a Building Object - Based Scale Determination Manager (BOBSDM) system routine according to an embodiment of the present disclosure.

[0009] Figure 7 Shows an exemplary embodiment of a flowchart of a building information access system routine according to an embodiment of the present disclosure. Detailed Embodiments

[0010] The present disclosure describes techniques for performing automated operations using a computing device, the automated operations involving analyzing visual data of an image acquired at a building to determine building information including building dimensions, and subsequently using the determined building information in one or more automated ways to provide navigation data for the building based on the building dimensions (e.g., for controlling the navigation of a mobile device such as an autonomous vehicle in the building). The automated determination of building dimensions and other building information may include analyzing the visual data of the building image to determine an estimated camera height of one or more cameras based on the identified types of visible installed building objects during image capture and other scale information of the image, using the determined image scale information to further determine the resulting building dimensions (e.g., the length and height of walls visible in the image to determine room dimensions), and optionally associating the building dimension data with a floor plan generated from the image analysis, which in at least some embodiments may be used for multi-room buildings under construction (e.g., houses, office buildings, etc.) and may be generated from or otherwise associated with panoramic images or other images (e.g., perspective images) acquired at acquisition locations within and around the building (e.g., without or without using information about the distance from the image acquisition location to walls or other objects in the surrounding building from any depth sensors or other distance measuring devices). The determined building information of the building can also be used in various ways, such as improving the automated navigation of the building, for display or other presentation in a corresponding GUI (Graphical User Interface) on one or more client devices, to enable virtual navigation of the building, etc. The following includes additional details regarding the automated determination and use of building information from building image analysis, and in at least some embodiments, some or all of the techniques described herein may be performed via automated operations of a Building Object-Based Scale Determination Manager (“BOBSDM”) system, as further discussed below.

[0011] As described above, the automated operations of the BOBSDM system may include: analyzing visual data of an image acquired at a building to determine building dimensions and other building information. In at least some embodiments, the automated analysis of the visual data of the image acquired at the building includes: identifying one or more defined types of visible structural objects or other installed objects at the building, the visible structural object or other installed object having a standardized size or otherwise having an expected size, such as a doorway (e.g., having an expected height of 80 inches or 2.03 meters), and / or a cabinet positioned on the floor (also referred to herein as a "floor cabinet") (e.g., having an expected height of 36 inches). In other embodiments, other object types may be used, each having one or a limited number of standardized sizes, whether in addition to or instead of doorways and / or floor cabinets, such as one or more of the following: oven, dishwasher, electrical panel, air outlet, fan blade, bed, television, power outlet, wall switch plate, etc. As a non-exclusive example, the four corners of a doorway may be identified in the image, and the corresponding coordinates of those points within the image may be converted to real-world coordinates using the expected height of 80 inches along the side between the two diagonal corners of the doorway, and then the estimated height of the camera used to acquire the image may be determined. For example, for a panoramic image in a straightened form (having vertical objects in the actual environment, such as the sides of a typical rectangular doorframe or the typical boundary between two adjacent walls, shown vertically in the image, such as in a single pixel column) and shown in an equirectangular format (e.g., having straight vertical object data remaining straight and having straight horizontal data, such as the top of a typical rectangular doorframe or the boundary between a wall and the floor remaining straight at the horizontal midline of the image, but gradually curving in a bulging manner relative to the horizontal midline in the equirectangular projection image as the distance from the horizontal midline in the image increases), the following formula (1) may be used to perform the determination of the estimated camera height h1 (in inches):

[0012] (1) 80 inches divided by

[0013] Where x1 is the horizontal (left - right) position of the lower - left corner of the doorway, x2 is the horizontal position of the lower - right corner of the doorway, x3 is the horizontal position of the upper - left corner of the doorway, x4 is the horizontal position of the upper - right corner of the doorway, z1 is the vertical (up - down) position of the lower - left corner of the doorway, z2 is the vertical position of the lower - right corner of the doorway, z3 is the vertical position of the upper - left corner of the doorway, and z4 is the vertical position of the upper - right corner of the doorway. For images that are not in the equirectangular format (e.g., perspective images in a straight - line format), a similar estimation of the camera height can be performed in a simplified manner, where formula (1) is adjusted to account for the lack of curvature of the horizontal data in the perspective image in a straight - line format. For one or more doorway objects identified in additional building images (e.g., the same doorway seen in different images, different doorways seen in the same image, different doorways seen in different images, etc.), such an estimated camera height can be determined similarly, such as for multiple building images taken of a building using the same constant actual camera height above a floor or other underlying surface (e.g., multiple images taken by one or more cameras during an image - acquisition session, where the one or more cameras are at a fixed height using the same tripod or held by an operator user at a consistent height), and an estimated doorway - based camera height can be determined from one or more individual camera - height estimates of one or more identified doorways (e.g., the total estimated camera height of multiple identified doorways, such as using the average or other mean of the individual camera - height estimates).

[0014] Additionally, an estimated camera height can be determined similarly based on one or more identified floor cabinets. As a non - exclusive example, for a floor cabinet identified in a straightened format in an image, and for one or some or all of the pixel columns of the visual data in the image that include the floor cabinet, the bottom of the cabinet (e.g., the intersection with the floor) and the top of the cabinet (e.g., the top of the countertop or other upper surface of the floor cabinet) are identified. Then, using the expected height of 36 inches between the rows of the image showing the bottom and top of the cabinet, the corresponding coordinates in the pixel columns of those rows can be converted to real - world coordinates, and if the image is in the equirectangular format, an estimated height of the camera used to acquire the image can be determined for the image (e.g., a panoramic image) in a manner similar to that discussed above with respect to formula (1), or in a manner adjusted for the lack of curvature of the horizontal data in an image in a non - equirectangular format. As a non - exclusive example of a straightened panoramic image with visual data of a floor cabinet spanning a sequence of pixel columns, the following formula (2) can be used to perform the determination of the estimated camera height h2 (in inches) for each such pixel column:

[0015] (2) 36 inches divided by (1 - tan(α2) / tan(α1))

[0016] Wherein, α1 is the angle between the line from the camera center to the front edge of the top of the cabinet of the pixel column and the line directly downward from the camera center in the direction of gravity, and α2 is the angle between the line from the camera center to the front edge of the bottom of the cabinet of the pixel column and the line directly downward from the camera center in the direction of gravity. Such an estimated camera height can be similarly determined for other identified floor-standing cabinet objects in one or more building images (e.g., the same cabinet object seen in different images, different cabinet objects seen in the same image, different cabinet objects seen in different images, etc.), such as for multiple building images taken of a building using the same constant actual camera height above the floor or other underlying surface (e.g., multiple images taken by one or more cameras in an image acquisition session, the one or more cameras using the same tripod at a fixed height or held by an operator user at a consistent height), and the estimated camera height of the total cabinet can be used to estimate the individual camera height from the one or more identified floor-standing cabinets (e.g., based on multiple individual pixel-column camera height estimates for a single floor-standing cabinet in a single panoramic image and / or based on pixel-column camera height estimates for multiple identified floor-standing cabinets in one or more panoramic images, such as using the average or other average of multiple individual camera height estimates).

[0017] After determining such doorway-based estimated camera height and cabinet-based estimated camera height for one or more building images acquired at a building (or based on one or more other camera height estimates for one or more other object types, whether in addition to or in place of doorway-based / or cabinet-based estimated camera height), information from the multiple estimated camera heights can be used to determine a final estimated camera height for use with the one or more images. In at least some embodiments and scenarios, the determination of the final estimated camera height for one or more images can include comparing the multiple estimated camera heights to determine if they meet one or more defined verification criteria (e.g., differing from each other by less than a defined maximum amount, such as a fixed distance, or percentage, or other relative difference between the minimum and maximum of the estimated camera heights, etc.), and if so, then determining the final estimated camera height for the one or more images based on the multiple estimated camera heights (e.g., using the average or other average, such as a weighted average based on the object type and the number of instances of the object; selecting the minimum or maximum or other representative value, etc.). In other embodiments and scenarios, the final camera height for a set of one or more building images can instead be determined using a single type of object visible in the one or more images.

[0018] Once the final camera height for one or more images of a building has been determined, this camera height data can be further used to determine additional scale information for objects and other elements visible in the images, such as building dimensions at least partially based on the width / length and / or height of walls visible in one or more images, the width of doorways, and other non-doorway wall openings visible in one or more images, including determining the actual dimensions of rooms and / or other areas on a floor plan (rather than just determining relative room shape dimensions and positions). Additionally, such determined camera height data and / or the resulting building dimensions can be used in other ways in other embodiments, whether in addition to or instead of determining the actual dimension information of a floor plan, such as determining the actual dimensions of structural and / or non-structural objects (e.g., for remodeling, such as determining the dimensions of an object to be replaced, the dimensions of empty areas for adding objects, etc.). The following includes additional details related to using the determined final camera height data in various ways, including determining building dimensions and other building information.

[0019] Additionally, in at least some embodiments and scenarios, one or more other techniques can be used to determine the final camera height and / or the resulting building dimensions for a set of one or more building images, in order to further verify the final camera height determined using one or more types of visible objects (e.g., to confirm that the camera heights from multiple techniques differ from each other by less than a defined maximum amount), or instead of using one or more types of visible objects, use one or more other techniques (e.g., if the estimated camera height data using one or more types of visible objects does not meet one or more verification criteria). As a non-exclusive example, an image acquired at the building can be used to generate a floor plan for the building, the exterior of the floor plan can be adapted to the exterior of the building as shown in an aerial image of the building (e.g., an image from a drone, an airplane, a satellite, etc.), and the adapted floor plan and information about the dimensions of the building exterior in the aerial image (e.g., using GPS data points associated with the aerial image) can be used to determine room dimensions and other building information (including the final camera height for one or more images acquired at the building). The following includes additional details related to using the identified structural building objects to determine the final camera height of a building image.

[0020] The described techniques provide various benefits in various embodiments, including allowing for more efficient and rapid identification and use of floor plans of multi-chamber buildings and other structures in ways that were previously not available, including by analyzing visual data of images acquired at the building to identify one or more defined types of objects to automatically determine the building dimensions of the building and using the standardized dimensions of these objects or otherwise expected dimensions to determine the camera height and other scaling information of the image. Additionally, such automated techniques allow for determining building dimensions and other building information using information acquired from an actual building environment (as opposed to plans regarding how the building should be constructed theoretically), and allow for capturing changes to structural elements and / or visual appearance elements that occur after the building was initially constructed. The described techniques also provide the benefit of allowing for improved automated navigation of a building by a mobile device (e.g., a semi-autonomous or fully autonomous vehicle) at least in part based on the building dimensions, including significantly reducing the computational power and time required to otherwise attempt to learn the building layout. Additionally, in some embodiments, the described techniques can be used to provide an improved GUI where a user can more accurately and rapidly obtain and use building information, which includes building dimensions (e.g., for navigating the interior of one or more buildings), including in response to a search request, as part of providing personalized information to the user, as part of providing value estimates and / or other information about the building to the user (e.g., after analyzing information of one or more target building floor plans that are similar to or otherwise match specified criteria to one or more initial floor plans), etc. Various other benefits are also provided by the techniques, some of which are further described elsewhere herein.

[0021] As described above, the automated operation of the BOBSDM system can include determining information for a building floor plan in at least some embodiments. Such a floor plan of a building can include a 2D (two-dimensional) representation of various information about the building (e.g., rooms, doorways between rooms and other inter-room connections, exterior doorways, windows, etc.) and can further be associated with various types of supplementary or additional information about the building (e.g., data of multiple other building-related attributes). Such additional building information can include, for example, one or more of the following: a 3D or three-dimensional model of the building including height information (e.g., for building walls and other vertical areas); a 2.5D or two and a half dimensional model of the building that includes a visual representation of walls and / or other vertical surfaces when reproduced without explicitly modeling the measured height of those walls and / or other vertical surfaces; images and / or other types of data captured in the rooms of the building, including panoramic images (e.g., 360° panoramic images), etc., as discussed in more detail below.

[0022] Additionally, in at least some embodiments and situations, some or all of the images obtained for a building and associated with the floor plan of the building may be panoramic images, each panoramic image being obtained at one of a plurality of acquisition locations within or around the building, such that a panoramic image is generated at each such acquisition location from one or more videos (e.g., 360° videos obtained from a smart phone or other mobile device held and turned by a user at that acquisition location), or from a plurality of images obtained in a plurality of directions from the acquisition location (e.g., from a smart phone or other mobile device held and turned by a user at that acquisition location), or by capturing all image information simultaneously (e.g., using one or more fish-eye lenses), etc. Such images may include visual data, and in at least some embodiments and situations, acquisition metadata regarding the acquisition of such panoramic images may be obtained and used in various ways, such as data obtained from an IMU (Inertial Measurement Unit) sensor or other sensors of the mobile device (e.g., compass heading data, GPS location data, etc.) when the mobile device is carried by the user or otherwise moved between acquisition locations. It should be understood that such panoramic images may in some cases be represented in a spherical coordinate system and provide up to 360° coverage around a horizontal and / or vertical axis, such that a user viewing the starting panoramic image may move the viewing direction within the starting panoramic image to a different direction, such that different images (or "views") are presented within the starting panoramic image (including, if the panoramic image is represented in a spherical coordinate system, converting the image being rendered to a planar coordinate system). Additional details regarding the acquisition and use of panoramic images or other images for a building are included below, including regarding Figures 2A to 2C and related descriptions.

[0023] In at least some embodiments, the BOBSDM system can operate in conjunction with one or more separate ICA (Image Capture and Analysis) systems and / or MIGM (Mapping Information and Generation Manager) systems, such as obtaining and using images from the ICA system and / or obtaining floor plans and other relevant information of a building from the MIGM system. In other embodiments, such a BOBSDM system can incorporate some or all of the functions of such ICA and / or MIGM systems as part of the BOBSDM system. In other embodiments, the BOBSDM system can operate without using some or all of the functions of the ICA and / or MIGM systems. For example, if the BOBSDM system obtains information about images and / or floor plans and relevant information of a building from other sources (e.g., by one or more users manually taking one or more such images, by one or more users manually creating or providing such floor plans and / or relevant information, etc.).

[0024] Regarding the functionality of such an ICA system, it can perform automated operations in at least some embodiments to acquire images (e.g., panoramic images) at various acquisition locations associated with a building (e.g., inside multiple rooms of a building), and optionally further acquire metadata related to the image acquisition process (e.g., compass heading data, GPS location data, etc.) and / or metadata related to the movement of the imaging device between the acquisition locations. In at least some embodiments, such acquisition and subsequent use of the acquired information can be performed without having or using information from depth sensors or other distance measurement devices regarding the distance from the image acquisition location to walls or other objects in the surrounding building or other structures. For example, in at least some such embodiments, such techniques can include using one or more mobile devices (e.g., cameras having one or more fish-eye lenses and mounted on a rotatable tripod or otherwise having an automatic rotation mechanism; cameras having one or more fish-eye lenses sufficient to horizontally capture 360 degrees without rotation; smart phones held by a user relative to the user at a constant position (e.g., chest height, eye height, etc.) and moved by the user so as to rotate the user's body and the held smart phone in a 360° circle about a vertical axis; cameras held or mounted on a user or the user's clothing; cameras mounted on air- and / or ground-based drones or other robotic devices, etc.) to capture visual data from a series of multiple acquisition locations within multiple rooms of a house (or other building). Additional details regarding the operation of one or more devices implementing the ICA system are included elsewhere in this document in order to perform such automated operations and, in some cases, further interact with one or more ICA system operator users in one or more ways to provide further functionality.

[0025] Regarding the functionality of such a MIGM system, it can perform automated operations in at least some embodiments to analyze a plurality of 360° panoramic images (and optionally other images) that have been acquired for the interior of a building (and optionally the exterior of the building), and determine the room shapes and the locations of the passageways connecting the rooms for some or all of these panoramic images, and in at least some embodiments and scenarios determine the wall elements and other elements of some or all of the rooms of the building. The types of connecting passageways between two or more rooms can include one or more doorway openings and other non-doorway wall openings between rooms, windows, stairways, non-room corridors, etc., and the automated analysis of the images can at least partially identify such elements based on identifying the outlines of the passageways, identifying what is inside the passageways that is different from their exteriors (e.g., different colors or shadows), etc. Different colors or shadows), etc. The automated operations can also include using the determined information to generate a floor plan of the building, and optionally generate other mapping information of the building, such as by using the inter-room passageway information and other information to determine the relative positions of the relevant room shapes with respect to each other, and optionally adding distance scaling information and / or various other types of information to the generated floor plan. Additionally, in at least some embodiments, the MIGM system can perform further automated operations to determine additional information and associate it with the building floor plan and / or specific rooms or locations within the floor plan in order to analyze images taken inside the building and / or other environmental information (e.g., audio) to determine specific attributes (e.g., color and / or material type and / or other characteristics of specific features or other elements such as floors, walls, ceilings, countertops, furniture, fixtures, household appliances, cabinets, islands, wall ovens, etc.); the presence and / or absence of specific features or other elements, etc.), or otherwise determine relevant attributes (e.g., the direction that a building feature or other element (e.g., a window) faces; the view from a specific window or other location, etc.). The following includes additional details regarding the operation of the (one or more) computing devices implementing the MIGM system in order to perform such automated operations and, in some cases, further interact with one or more MIGM system operator users in one or more ways to provide further functionality.

[0026] In some embodiments and scenarios, an adjacency graph representing information about room adjacencies in a building may also be generated, and the adjacency graph also stores or otherwise includes some or all such data of the building, such as by analyzing a floor plan, and at least some of such data is stored in or associated with the nodes of the adjacency graph, where the nodes of the adjacency graph represent some or all of the rooms in the floor plan (e.g., each node contains information about room attributes represented by the node), and / or at least some of such attribute data is stored in or otherwise associated with the edges between the nodes, where the edges between the nodes represent connections between adjacent rooms via doorways or other non-doorway openings in the room-to-room walls, or in some cases further represent adjacent rooms that share at least a portion and optionally all of a wall without any direct room-to-room opening connecting the two rooms (e.g., each edge contains information about the connection status between the rooms represented by the nodes interconnected by the edge, such as whether there is a room-to-room opening between two rooms, and / or a type of room-to-room opening or other type of adjacency between two rooms, such as no direct room-to-room wall opening connecting). In some embodiments and scenarios, the floor plan and / or the adjacency graph may further represent at least some information outside the building, such as an external area adjacent to a doorway or other wall openings between the building and external and / or other accessory structures located on the same property as the building (e.g., a garage, shed, pool house, detached guest house, mother-in-law unit or other accessory residential unit, pool, patio, deck, sidewalk, garden, courtyard, etc.), or more generally, some or all of the external areas of a property including one or more buildings (e.g., a house and one or more accessory buildings or other accessory structures). Such external areas and / or other structures may be represented in the adjacency graph and / or on the floor plan in various ways, such as by a separate node in the adjacency graph for each such external area or other structure or by arranging the floor plan on a rendering of the property including the external area, or as attribute information associated with the corresponding nodes or edges of the adjacency graph, or instead as attribute information associated with the adjacency graph as a whole (for the building as a whole). In at least some embodiments, the adjacency graph may also have associated attribute information for the corresponding rooms and room-to-room connections in order to represent some or all of the information available on the floor plan and otherwise associated with the floor plan (or in some examples and scenarios, information represented in and associated with a 3D model of the building) within the adjacency graph. For example, if there is an image associated with a particular room in the floor plan or other associated area (e.g., an external area), the corresponding visual attribute may be included within the adjacency graph, either as part of the associated room or other area, or as a separate node layer within the graph representing the image.In embodiments with adjacency information in a form other than an adjacency graph, some or all of the above types of information may be stored in or otherwise associated with the adjacency information, including information about rooms, information about adjacencies between rooms, information about the connection status between adjacent rooms, information about the attributes of a building, etc. Additional details regarding the generation and use of floor plans and / or adjacency graphs are included below, including examples thereof and related descriptions. Figures 2D to 2H and its related description.

[0027] Additionally, the automated operation of the BOBSDM system may also include generating and using one or more vector-based embeddings (also referred to herein as "vector embeddings") to concisely represent information in the adjacency graph of a building's floor plan, so as to summarize the semantic meaning and spatial relationships of the floor plan in a way that allows some or all of the floor plan to be reconstructed from the vector embedding. In various embodiments, such vector embeddings can be generated in various ways, for example by using representation learning and one or more trained machine learning models, and in at least some such embodiments, such vector embeddings can be encoded in a format that is not easily discernible by a human reader. Non-exclusive examples of techniques for generating such vector embeddings are included in the following documents, which are hereby incorporated by reference in their entirety: "Symmetric Graph Convolution Autoencoder For Unsupervised Graph Representation Learning" by Jiwoong Park et al., 2019 International Conference on Computer Vision, August 7, 2019; "Inductive Representation Learning On Large Graphs" by William L Hamilton et al., 31st Conference on Neural Information Processing Systems, June 7, 2017; and "Variational Graph Auto-Encoders" by Thomas N. Kipf et al., 30th Conference on Neural Information Processing Systems (Bayesian Deep Learning Workshop), November 21, 2016.

[0028] As described above, a floor plan may have various information associated with individual rooms and / or connections between rooms and / or corresponding buildings and / or the property as a whole, and the corresponding adjacency graph and / or vector embedding for such a floor plan may include some or all of such associated information (e.g., attributes of nodes representing rooms in the adjacency graph and / or attributes of edges representing connections between rooms in the adjacency graph and / or attributes representing the adjacency graph as a whole, e.g., in a node representing an entire building, and having corresponding information encoded in the associated (multiple) vector embeddings). Such associated information may include various types of data, including information about one or more of the following non-exclusive examples: room type, room size, locations of windows and doors, and other inter-room openings in the room, room shape, view type of each external window, information and / or copies of images taken in the room, information and / or copies of audio or other data taken in the room, various types of information about the features of one or more rooms (e.g., automatically recognized based on image analysis, provided by an operator user of the BOBSDM system and / or by an end user who views information about the floor plan and / or by an operator user of the ICA and / or MIGM systems as part of taking information about the building and generating a floor plan for the building, etc.), attributes of structures and objects (e.g., color, shape, material, age, condition, quality, etc.), type of connection between rooms, size of connection between rooms, etc. Additionally, in at least some embodiments, one or more additional subjective attributes may be determined for the floor plan and associated with the floor plan, e.g., by analyzing the floor plan information (e.g., the adjacency graph of the floor plan) through one or more trained machine learning models (e.g., classification neural network models) to identify the floor plan features of a building as a whole or a particular building floor (e.g., open floor plan; typical / normal vs. atypical / odd / unusual floor plan; standard vs. non-standard floor plan; accessibility-friendly floor plan, e.g., being accessible in terms of one or more characteristics such as disability and / or advanced age, etc.). In at least some such embodiments, one or more classification neural network models are part of the BOBSDM system and are trained through supervised learning using labeled data that identifies floor plans having each of the possible features, while in other embodiments, such classification neural network models may instead use unsupervised clustering. Additional details regarding determining and using the attribute information of floor plans are included below, including examples of Figures 2D to 2H and their related descriptions.

[0029] After generating an adjacency graph and one or more vector embeddings for a floor plan of a building and associated information, the generated information can be used by the BOBSDM system as a specified standard to automatically determine one or more other similar or otherwise matching floor plans of other buildings in various ways in various embodiments. For example, in some embodiments, an initial floor plan is identified, and one or more corresponding vector embeddings for the initial floor plan are generated and compared with the generated vector embeddings for other candidate floor plans to determine the differences between the vector embeddings of the initial floor plan and the vector embeddings of some or all of the candidate floor plans. A smaller difference between two vector embeddings corresponds to a higher similarity between the building information represented by those vector embeddings. In various embodiments, the differences between two such vector embeddings can be determined in various ways, including, by way of non-exclusive example, by using one or more of the following distance metrics: Euclidean distance, cosine distance, graph edit distance, a custom distance metric specified by the user, etc.; and / or otherwise determining similarity without using such distance metrics. In at least some embodiments, multiple such initial floor plans can be identified and used in the described manner to determine a combined distance between a group of vector embeddings for the multiple initial floor plans and the vector embeddings for each of the multiple other candidate floor plans, for example, by determining the individual distances from each of the initial floor plans to a given other candidate floor plan and by combining the multiple individually determined distances in one or more ways (e.g., mean or other average, cumulative total, etc.) for the group of vector embeddings for the multiple initial floor plans to the given other candidate floor plan.

[0030] Additionally, in some embodiments, one or more explicitly specified criteria are received in addition to or instead of one or more initial floor plans, and one or more corresponding vector embeddings for each of a plurality of candidate floor plans are compared with information generated from the specified criteria to determine which candidate floor plans meet the specified criteria (e.g., match above a defined similarity threshold), such as by generating a representation of a building corresponding to the criteria (e.g., having attributes identified in the criteria), and generating one or more vector embeddings for the building representation for vector embedding comparison in the above manner. In various embodiments and scenarios, the specified criteria can be of various types, such as one or more of the following non-exclusive examples: search terms corresponding to specific attributes of rooms and / or connections between rooms and / or the building as a whole (objective attributes that can be independently verified and / or replicated, and / or subjective attributes determined by using a corresponding classification neural network); information identifying adjacency information between two or more rooms or other areas; information about views obtainable from windows or other external openings of the building; information about the orientation of windows or other structural features or other elements of the building (e.g., to determine natural lighting information obtainable via those windows or other structural elements, optionally on a specified date and / or season and / or time), etc. Non-exclusive illustrative examples of such specified criteria include the following: a bathroom adjacent to a bedroom (e.g., without an intervening hall or other room); a deck adjacent to a family room (optionally having a specific type of connection, such as French doors); two bedrooms facing south; a kitchen having a tiled island and a view facing north; a master bedroom having a view of the ocean or more generally a view of water; any combination of such specified criteria, etc. Additionally, in some embodiments, one or more target floor plans similar to the specified criteria associated with a particular end user are identified (e.g., based on one or more initial target floor plans selected by the end user and / or identified as previously of interest to the end user, whether such floor plans are specified based on explicit and / or implicit activities of the end user; based on one or more search criteria specified by the end user, whether explicitly and / or implicitly, etc.), and are used in further automated activities to personalize the interaction with the end user. In various embodiments, such further automated personalized interaction can be various types of interactions, and in some embodiments, can include displaying or otherwise presenting information about one or more target floor plans and / or additional associated information to the end user.

[0031] In some embodiments, the described techniques may further include additional operations. For example, in at least some embodiments, machine learning techniques may be used to learn the properties and / or other features of the adjacency graph for encoding in the corresponding vector embeddings generated, such as to be able to subsequently automatically identify the properties and / or other features of a floor plan of a building having properties that meet a target criterion (e.g., number of bedrooms; number of bathrooms; connectivity between rooms; dimensions and / or size of each room; number of windows / doors in each room; type of view obtainable from an external window, such as water, mountains, backyard, or other exterior areas of the property, etc.; location of windows / doors in each room, etc.). Additionally, in at least some embodiments, machine learning techniques may be used to identify one or more types of objects in one or more images (e.g., doorways, cabinets, etc.), such as via one or more machine learning models trained to determine such information (e.g., machine learning models for each defined object type). Additionally, in at least some embodiments, machine learning techniques may be used to determine the estimated camera height of one or more images, such as via one or more machine learning models trained to determine such information (e.g., machine learning models for each defined object type).

[0032] For purposes of illustration, some embodiments are described below in which a particular type of information is targeted to a particular type of structure and obtained, used, and / or presented in a particular manner by using a particular type of device. However, it will be understood that the techniques described can be used in other ways in other embodiments, and thus the present invention is not limited to the provided exemplary details. As a non-exclusive example, although in some embodiments a particular type of data structure (e.g., floor plan, adjacency graph, vector embedding, etc.) is generated and used in a particular manner, it should be understood that in other embodiments other types of information describing floor plans and other associated information can be similarly generated and used, including for buildings (or other structures or layouts) separate from a house, and in other embodiments, floor plans identified as matching specified criteria can be used in other ways. Additionally, the term "building" as used herein refers to any partially or fully enclosed structure, typically but not necessarily including one or more rooms that visibly or otherwise divide the interior space of the structure. Non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, condominiums, office buildings, commercial buildings or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.), supplementary structures on a property having another main building (e.g., a detached garage or a shed on a property with a house), etc. The term "acquire" or "capture" as used herein with respect to the interior of a building, an acquisition location, or other location (unless the context clearly indicates otherwise) can refer to any recording, storage, or logging of media, sensor data, and / or other information related to spatial characteristics and / or visual characteristics and / or otherwise perceivable characteristics of the interior of a building or a subset thereof, such as by a recording device or by another device that receives information from the recording device. As used herein, the term "panoramic image" can refer to a visual representation that is based on, includes, or can be divided into a plurality of discrete component images, the plurality of discrete component images originating from a substantially similar physical location in different directions and depicting a larger field of view than any of the discrete component images depicted individually, including images having a wide enough angular view from the physical location to include an angle beyond what is perceptible from a person's gaze in a single direction. The term "series" of acquisition locations as used herein generally refers to two or more acquisition locations, each of which is visited at least once in a corresponding order, regardless of whether other non-acquisition locations are visited between them, and regardless of whether the visits to the acquisition locations occur during a single continuous time period, or at multiple different times, or by a single user and / or device, or by multiple different users and / or devices. Additionally, for purposes of illustration, various details are provided in the drawings and the text, but are not intended to limit the scope of the present invention.For example, the dimensions and relative positions of the drawing elements need not be drawn to scale, where some details are omitted or provided with greater prominence (e.g., through sizing and positioning) to enhance legible clarity. Additionally, the same reference numerals may be used in the drawings to identify the same or similar elements or acts.

[0033] Figure 1 An example block diagram of various computing devices and systems that may participate in the described technology in some embodiments, such as with respect to an example building 198 (a house in this example) and an example building object-based scale determination manager ("BOBSDM") system 140 that is executed on one or more server computing systems 180 in this example embodiment.

[0034] Figure 1 An example block diagram of various computing devices and systems that may participate in the described technology in some embodiments, such as with respect to an example building 198 (a house in this example), and an example building object-based scale determination manager ("BOBSDM") system 140 that is executed on one or more server computing systems 180 in this example embodiment.

[0035] In the illustrated embodiment, the BOBSDM system 140 analyzes the obtained building images 141 of the building (e.g., some or all of the images 165 obtained by the ICA system 160) to determine the dimensional information 143 of the building (e.g., the dimensions of visible objects, rooms, etc.), such as by using visible objects 142 identified in images of one or more defined types (e.g., doorways, floor cabinets, etc.) and having a standardized or other expected height or other dimensions, in order to estimate the camera height of those images during the capture of one or more camera devices and optionally other image scale information 143. The BOBSDM system may also use the determined building dimensional information in various ways, including determining the dimensions of rooms and / or the building and associating them as a whole with the floor plan of the building, such as for improved navigation of the building. In some embodiments and scenarios, the BOBSDM system can optionally further use support information provided by the system operator user via the computing device 105 through one or more intervening computer networks 170, and in some embodiments and scenarios, some or all of the determinations performed by the BOBSDM system may include using one or more trained machine learning models (e.g., one or more trained neural networks). In some embodiments, the building images 141 analyzed by the BOBSDM system may be obtained in a different manner than via the ICA and / or MIGM system 160 (e.g., if such ICA and / or MIGM system is not part of the BOBSDM system), such as receiving the building images from other sources. Other details regarding the automated operation of the BOBSDM system are included elsewhere herein, including regarding Figures 2I to 2L and Figure 6 .

[0036] Additionally, an Interior Capture and Analysis (“ICA”) system (e.g., an ICA system 160 executed on one or more server computing systems 180 such as part of a BOBSDM system; an optional ICA system application 154 executed on a mobile image acquisition device 185, etc.) captures information 165 about one or more buildings or other structures (e.g., by capturing one or more 360° panoramic images and / or other images of a plurality of acquisition locations 210 in an exemplary house 198), and a MIGM (Mapping Information Generation Manager) system 160 (e.g., as part of a BOBSDM system) executed on one or more server computing systems 180 further uses the captured building information and optionally additional supporting information (e.g., provided by a system operator user via a computing device 105 through an intervening computer network 170) to generate and provide building floor plans 155 and / or other mapping-related information (not shown) for one or more buildings or one or more other structures. In the illustrated embodiment, the ICA and MIGM systems 160 operate as part of a BOBSDM system 140, which analyzes building images 141 (e.g., images 165 acquired by the ICA system) and generates and uses corresponding building information 144 (e.g., as part of a floor plan generated by the MIGM system) in one or more other automated ways, but in other embodiments, may operate separately from the BOBSDM system. Similarly, although in this exemplary embodiment the ICA and MIGM systems 160 are shown as being executed on the same server computing system 180 as the BOBSDM system (e.g., all systems are operated by a single entity or otherwise execute in cooperation with each other, such as integrating some or all of the functions of all systems together), in other embodiments, the ICA system 160 and / or the MIGM system 160 and / or the BOBSDM system 140 may operate on one or more other systems separate from system 180 (e.g., on a mobile device 185; one or more other computing systems not shown; etc.), whether as a replacement for copies of those systems executed on system 180 or in addition to copies of those systems executed on system 180 (e.g., such that a copy of the MIGM system 160 executed on the device 185 incrementally generates at least part of a building floor plan as the ICA system 160 executed on the device 185 and / or on this copy of the MIGM system captures building images, while another copy of the MIGM system optionally executes on one or more server computing systems to generate a final complete building floor plan after all images have been acquired), and in other embodiments, the BOBSDM may alternatively operate without the ICA system and / or the MIGM system and alternatively obtain panoramic images (or other images) and / or building floor plans from one or more external sources.Other details related to the automated operation of the ICA and MIGM systems are included elsewhere herein, including regarding. Figures 2A to 2H and regarding Figure 4 and Figures 5A to 5B .

[0037] The various components of the mobile computing device 185 are also shown in Figure 1 , including one or more hardware processors 132 (e.g., CPU, GPU, etc.) that execute software (e.g., optional ICA application 154, optional browser 162, etc.) using executable instructions stored and / or loaded on one or more memory / storage components 152 of the device 185, and optionally one or more imaging systems 135 of one or more types to obtain visual data of one or more panoramic images 165 and / or other images (not shown, such as stereoscopic images). In some embodiments, some or all such images may be provided by one or more separate associated camera devices 184 (e.g., via a wired / cable connection, via Bluetooth or other inter-device wireless communication, etc.), whether in addition to or in place of the images captured by the mobile device 185. The illustrated embodiment of the mobile device 185 also includes: one or more sensor modules 148, which in this example includes a gyroscope 148a, an accelerometer 148b, and a compass 148c (e.g., as part of one or more IMU units (not shown separately) on the mobile device); one or more control systems 147 that manage the I / O (input / output) and / or communication and / or networking of the device 185 (e.g., receive instructions from the user and present information to the user), such as for other device I / O and communication components 143 (e.g., network interface or other connection, keyboard, mouse or other pointing device, microphone, speaker, GPS receiver, etc.), a display system 149 (e.g., having a touch-sensitive screen), optionally one or more depth-sensing sensors or other ranging components 136 of one or more types, optionally a GPS (or Global Positioning System) sensor 134 or other position-determining sensor (not shown in this example), etc. Other computing devices / systems 105, 175, and 180 and / or camera devices 184 may include various hardware components and stored information in a manner similar to the mobile device 185, and for the sake of brevity, the various hardware components and stored information are not shown in this example and are discussed in more detail below with reference to Figure 3 .

[0038] One or more users (not shown) of one or more client computing devices 175 may also interact with the BOBSDM system 140 (and optionally the ICA system 160 and / or the MIGM system 160) via one or more computer networks 170 to obtain determined building dimension information and / or assist in determining building dimension information, and to obtain and use base images and / or resultant floor plans in one or more other automated ways. Such user interaction may include, for example, specifying target criteria for searching for corresponding floor plans or otherwise providing information to the user about target criteria of interest, or obtaining and optionally interacting with one or more specifically identified floor plans and / or with additional associated information (e.g., changing between a floor plan view and a view of a specific image at an acquisition location within or near the floor plan; changing the horizontal and / or vertical viewing direction of a corresponding view of a panoramic image to determine the portion of the panoramic image that the current user viewing direction is pointing to, etc.). Additionally, a floor plan (or a portion thereof) may be linked to one or more other types of information or otherwise associated with one or more other types of information, including for a multi-floor or multi-story building, multiple associated sub-floor plans for different floors or levels that are interconnected (e.g., via connecting stairwells), a two-dimensional ("2D") floor plan of a building that is to be linked to a three-dimensional ("3D") rendering of the building or otherwise associated with the three-dimensional ("3D") rendering of the building, etc. Additionally, although not shown in Figure 1 , in some embodiments, the client computing device 175 (or other device, not shown) may receive and use information about determined building storage and / or about identified floor plans or other mapping-related information in an additional way to control or assist in the automated navigation activities of those devices (e.g., autonomous vehicles or other devices), either in lieu of displaying the identified information or in addition to displaying the identified information.

[0039] In Figure 1 the computing environment shown, the network 170 may be one or more publicly accessible linked networks, possibly operated by various different parties (e.g., the Internet). In other implementations, the network 170 may have other forms. For example, the network 170 may instead be a private network, such as a company or university network that is not fully or partially accessible to non-privileged users. In other implementations, the network 170 may include a private network and a public network, where one or more private networks may be accessible to and / or accessed from one or more public networks. Additionally, the network 170 may include various types of wired and / or wireless networks in various scenarios. Additionally, the client computing device 175 and the server computing system 180 may include various hardware components and stored information, as discussed in more detail below with reference to Figure 3 more detail.

[0040] In Figure 1 the example of, the ICA system can perform automated operations, such as using visual data obtained via the mobile device 185 and / or the associated camera device 184, the automated operations involving generating a plurality of 360° panoramic images at a plurality of associated acquisition locations (e.g., in multiple rooms or other locations within a building or other structure, and optionally around some or all of the exterior of the building or other structure), and for generating and providing a representation of the interior of the building or other structure. For example, in at least some such embodiments, such techniques can include using one or more mobile devices (e.g., a camera having one or more fish-eye lenses and mounted on a rotatable tripod or otherwise having an automatic rotation mechanism, a camera having a fish-eye lens sufficient to horizontally shoot 360° without rotating, a smart phone held and moved by a user, a camera held by a user or mounted on a user or user's clothing, etc.) to capture data from a series of a plurality of acquisition locations within multiple rooms of a house (or other building), and optionally further capturing involving movement of the acquisition device (e.g., movement at the acquisition location, such as rotation; movement between some or all of the acquisition locations, such as for linking a plurality of acquisition locations together; etc.), in at least some cases, without having a measured distance between acquisition locations or other measured depth information to an object in the environment around the acquisition locations (e.g., without using any depth sensing sensors). After the information at the acquisition location is captured, the technique can include generating a 360° panoramic image having 360 degrees of horizontal information about a vertical axis from the acquisition location (e.g., a 360° panoramic image showing the surrounding room in an equirectangular format), and then providing the panoramic image for subsequent use by the MIGM and / or BOBSDM systems.

[0041] One or more end users (not shown) of one or more building information access client computing devices 175 can also interact with the BOBSDM system 140 (and optionally with the MIGM system 160 and / or the ICA system 160) via the computer network 170, in order to obtain, display the generated floor plan (and / or other generated mapping information) and / or associated images, and interact with the generated floor plan (and / or other generated mapping information) and / or the determined building structure attribute information and / or associated images (e.g., by providing information about one or more indicated buildings of interest and / or other criteria and receiving information about one or more corresponding matching buildings), as discussed in more detail elsewhere herein, including regarding Figure 7 . Additionally, although in Figure 1Although not shown, a floor plan (or a portion thereof) may be linked to or otherwise associated with one or more additional types of information, such as one or more associated and linked images or other associated and linked information, including, for example, a two-dimensional (“2D”) floor plan of a building being linked to or otherwise associated with a separate 2.5D model floor plan rendering of the building and / or a 3D model floor plan rendering of the building, and including, for a floor plan of a multi-story or other multi-level building, having a plurality of associated sub-floor plans for interconnecting (e.g., via a connecting stairwell) different floors or levels, or being part of a common 2.5D and / or 3D model. Thus, non-exclusive examples of an end user's interaction with a displayed or otherwise generated 2D floor plan of a building may include one or more of the following: changing between a view of a specific image at an acquisition location within or near the floor plan and the floor plan view; changing between a 2D floor plan view and a 2.5D or 3D model view, the 2.5D or 3D model view optionally including images texture-mapped to the walls of the displayed model; changing the horizontal and / or vertical viewing direction of a corresponding subset view (or an entrance into a panoramic image) of a displayed panoramic image in order to determine the portion of the panoramic image in a 3D coordinate system that the current user viewing direction is pointing to and rendering a corresponding planar image that shows that portion of the panoramic image without the curvature or other distortion that was present in the original panoramic image, etc. Additionally, although not shown in Figure 1 In some embodiments, the client computing device 175 (or other device, not shown) may receive and use the generated floor plan and / or other generated mapping-related information in an additional way to control or assist the automated navigation activities of those devices (e.g., an autonomous vehicle or other device), either in lieu of displaying the generated information or in addition to displaying the generated information.

[0042] Figure 1 Exemplary building interior environments are also described, in which 360° panoramic images and / or other images are acquired, such as via an ICA system, and are used by the BOBSDM and / or MIGM systems, respectively, to determine building dimension information and to generate and provide one or more corresponding building floor plans (e.g., multiple incremental partial building floor plans). In particular, Figure 1Shows a floor of a multi - storey building (or other building) 198 with an interior that is at least partially captured by a plurality of panoramic images, e.g., by a mobile image capture device 185 having image capture capabilities and / or one or more associated camera devices 184 as they move through the building interior to a series of multiple acquisition locations 210 (e.g., starting from acquisition location 210A, moving along travel path 115 to acquisition location 210B, etc., and ending at acquisition locations 210 - O or 210P outside the building). One implementation of the ICA system can automatically perform or assist in capturing data representing the building interior (and further analyzing the captured data to generate 360° panoramic images to provide a visual representation of the building interior), and one implementation of the MIGM system can analyze the visual data of the acquired images to generate one or more building floor plans for the building 198 (e.g., multiple incremental building floor plans, including determining the orientations between images, such as the orientations 215 - AB, 215 - AC, and 215 - BC between image acquisition locations 210A, 210B, and 210C). Although such a mobile image acquisition device can include various hardware components, such as cameras, one or more sensors (e.g., gyroscopes, accelerometers, compasses, etc., e.g., one or more IMUs of a mobile device, or parts of an inertial measurement unit; altimeters; light detectors; etc.), GPS receivers, one or more hardware processors, memories, displays, microphones, etc., in at least some implementations, the mobile device may not have access to or use a device to measure the depth of an object in the building relative to the location of the mobile device, such that in such implementations, the relationship between different panoramic images and their acquisition locations can be determined partially or entirely based on features in different images, but without using any data from any such depth sensors, while in other implementations, such depth data can be obtained and used. Additionally, although in Figure 1 direction indicator 109 is provided for reference by the reader in relation to the exemplary building 198, in at least some implementations, the mobile device and / or the ICA system may not use such absolute direction information and / or absolute location, such that in such implementations, the relative directions and distances between acquisition locations 210 are determined without considering the actual geographical location or orientation, while in other implementations, such absolute direction information and / or absolute location can be obtained and used.

[0043] In operation, the mobile device 185 and / or the camera device 184 reach a first acquisition location 210A within a first room inside the building (in this example, in the living room accessible via the outer door 190-1), and capture or acquire a view of the portion of the inside of the building visible from this acquisition location 210A (e.g., some or all of the first room, and optionally, a small portion of one or more other adjacent or neighboring rooms, such as through doorway wall openings, non-doorway wall openings, corridors, stairways, or other connection passages from the first room). The view capture can be performed in various ways as discussed herein and can include multiple objects or other features (e.g., structural details) visible in the image captured from the acquisition location. In Figure 1 this example, these objects or other features within the building 198 include doorways 190 (including 190-1 to 190-6, such as those with swinging and / or sliding doors), windows 196 (including 196-1 to 196-8), corners or edges 195 (including corner 195-1 at the northwest corner of the building 198, corner 195-2 at the northeast corner of the first room, corner 195-3 at the southwest corner of the first room, corner 195-4 at the southeast corner of the first room, corner 195-5 at the northern edge of the inter-room passage between the first room and the corridor, etc.), furniture 191 to 193 (e.g., recliner 191; chair 192; table 193, etc.), pictures or paintings or televisions or other hanging objects 194 (e.g., 194-1 and 194-2) hanging on the walls, light fixtures ( Figure 1 not shown in the figure), various built-in household appliances or light fixtures or other structural elements ( Figure 1 not shown in the figure), etc. The user may also optionally provide a text or auditory identifier to be associated with the acquisition location and / or the surrounding rooms (e.g., "living room" for one of the acquisition locations 210A or 210B or for the room including the acquisition locations 210A and / or 210B), while in other embodiments, the ICA and / or MIGM system can automatically generate such an identifier (e.g., by automatically analyzing the images and / or videos and / or other recorded information of the building to perform the corresponding automatic determination, such as by using machine learning; at least partially based on input from the ICA and / or MIGM system operator user, etc.) or may not use an identifier.

[0044] After the first acquisition location 210A has been captured, the mobile device 185 and / or the camera device 184 may move or move to the next acquisition location (such as acquisition location 210B) during the movement between acquisition locations, optionally recording images and / or videos and / or other data from hardware components (e.g., from one or more IMUs, from the camera, etc.). At the next acquisition location, the mobile device 185 and / or the camera device 184 may similarly capture 360° panoramic images and / or other types of images from that acquisition location. For some or all of the rooms in a building and in some cases outside the building, this process may be repeated, as shown for additional acquisition locations 210C - 210P in this example, where images from acquisition locations 210A to 210 - O are acquired during a single image acquisition session (e.g., in a substantially continuous manner, such as within a total of 5 minutes or 15 minutes), and images from acquisition location 210P are optionally acquired at different times (e.g., from a street adjacent to the building or the front yard of the building). In this example, multiple acquisition locations 210K - 210P are outside the building 198 but associated with the building 198, including acquisition locations 210L and 210M in one or more additional structures 189 on the same property 241 (e.g., ADU or accessory dwelling unit; garage; shed; etc.), acquisition location 210K on an exterior platform or terrace 186, and acquisition locations 210N - 210P at multiple yard locations on the property (e.g., backyard 187, side yard 188, front yard including acquisition location 210P, etc.). The acquired images for each acquisition location may be further analyzed, including in some embodiments rendering each panoramic image in an equirectangular format or otherwise placing it, either at the time of image acquisition or later, and further analyzing it by the MIGM and / or BOBSDM systems in the manner described herein.

[0045] Reference Figure 1 Various details are provided, but it should be understood that the details provided are non - exclusive examples included for illustrative purposes and that other embodiments may be implemented in other ways without some or all of such details.

[0046] Figures 2A to 2L An example of automatically analyzing visual data of images acquired at a building to determine building information, the building information including building dimensions, and then using such information in one or more automated ways, such as Figure 1 the building 198 discussed in

[0047] In particular, Figure 2A Example image 250a is shown, for example, from Figure 1A non-panoramic stereoscopic image taken in the northeast direction starting from the acquisition location 210B in the living room of the house 198 (or a subset view facing the northeast direction of a 360° panoramic image taken from this acquisition location and formatted in a straight line). In this example, a direction indicator 109a is further shown to indicate the direction in the northeast where the image is taken. In the example shown, the displayed image includes built-in elements (e.g., the lamp 130a), furniture (e.g., the chair 192-1), two windows 196-1, and a picture 194-1 hanging on the north wall of the living room. No room-to-room passages (e.g., doorways or other wall openings) entering or leaving the living room are visible in this image. However, in the image 250a, multiple room boundaries are visible, including the horizontal boundary between the visible part of the north wall of the living room and the ceiling and floor of the living room, the horizontal boundary between the visible part of the east wall of the living room and the ceiling and floor of the living room, and the vertical wall-to-wall boundary 195-2 between the north wall and the east wall.

[0048] Figure 2B Continue Figure 2A Example, and shows two additional stereograms, including Figure 1 An additional stereoscopic image 250b taken in the northwest direction from the acquisition location 210B in the living room of the house 198. A direction indicator 109b is further shown to indicate the northwest direction where the image is taken. In this example image, a small part of one of the windows 196-1, part of the window 196-2, and new lighting equipment 130b continue to be visible. Additionally, horizontal and vertical room boundaries are visible in the image 250b in a similar manner to Figure 2A That way.

[0049] Figure 2B Also shown is Figure 1 A third stereoscopic image 250c taken in the southwest direction in the living room of the house 198, e.g., taken from the acquisition location 210B. A direction indicator 109c is further shown to indicate the southwest direction where the image is taken. In this example image, part of the window 196-2 continues to be visible, and the chaise longue 191 and visible horizontal and vertical room boundaries are visible in a similar manner to Figure 2A And Figure 2B Two room-to-room passages for the living room are also shown in this example, which in this example include a doorway 190-1 with a swinging door for entering and leaving the living room ( Figure 1 Identifying it as a door to the outside of the house, e.g., the front yard), and a doorway 190-6 with a sliding door for moving between the living room and the side yard 188. As Figure 1as shown by the information in. Additionally, there are other non-doorway wall openings 263a in the east wall of the living room for movement between the living room and the corridor, but they are not visible in images 250a to 250c. It should be understood that various other stereoscopic images can be obtained from acquisition location 210B and / or other acquisition locations and displayed in a similar manner.

[0050] Figure 2C Continue Figures 2A - 2B as an example, and shows a 360° panoramic image 255c (e.g., obtained from acquisition location 210B), which displays the entire living room in equirectangular format. Since the panoramic image does not have an orientation in the same way as Figures 2A - 2B the stereoscopic image, the orientation indicator 109 is not shown in Figure 2C although the pose of the panoramic image may include one or more associated orientations (e.g., the start and / or end orientations of the panoramic image, e.g., if obtained by rotation). A portion of the visual data of the panoramic image 255c corresponds to the first stereoscopic image 250a (shown approximately in the central portion of image 250d), while the left portion of image 255c and the far-right portion of image 255c contain visual data corresponding to the visual data of stereoscopic images 250b and 250c. This exemplary panoramic image 255d includes windows 196-1, 196-2, and 196-3, furniture 191-193, doorways 190-1 and 190-6, and a non-doorway wall opening 263a leading to the corridor room (this opening shows a portion of door 190-3 visible in the adjacent corridor). Image 255c also shows various room boundaries in a manner similar to the stereoscopic image, but the horizontal boundaries are displayed in a gradually curved manner as the distance from the horizontal midline of the image increases. The visible boundaries include vertical wall-to-wall boundaries 195-1 to 195-4, a vertical boundary 195-5 at the left / north side of the corridor opening, a vertical boundary at the right side of the corridor opening, and horizontal boundaries between the wall and the floor and between the wall and the ceiling.

[0051] Figure 2C An example of a 2D floor plan 230c for house 198 is also shown, which can be generated by the MIGM system and presented to the end user in GUI260c, where the living room is the westernmost room of the house (as reflected by the orientation indicator 209). It should be understood that in some embodiments, a 3D or 2.5D floor plan building model with rendered wall height information can be generated and displayed similarly. Whether in addition to or in place of such a 2D floor plan, an example of such a floor plan building model is shown in Figure 2HShown in. In this example, various types of information are shown on the 2D floor plan 230c. For example, this type of information may include one or more of the following: room labels added to some or all of the rooms (e.g., "Living Room" for the living room); room dimensions added for some or all of the rooms, such as based on building dimension information determined by the BOBSDM system; visual indications of features, such as installed fixtures or household appliances (e.g., kitchen household appliances, bathroom items, etc.) or other built-in elements added for some or all of the rooms (e.g., kitchen island); visual indications added for some or all of the rooms for additional types of associated and linked information (e.g., other panoramic images and / or stereoscopic images that the end user can select for further display, audio annotations and / or sound recordings that the end user can select for further presentation, etc.); visual indications added for some or all of the rooms for structural features such as doors and windows; visual indications of visual appearance information (e.g., the color and / or material type and / or texture of installed items, such as floor coverings or wall coverings or surface coverings); visual indications of views from a particular window or other building location and / or other information about the exterior of the building (e.g., the type of exterior space; items present in the exterior space; other relevant buildings or structures, such as sheds, garages, pools, patios, courtyards, walkways, gardens, etc.); keywords or legends 269 that identify visual indicators for one or more types of information. When shown as part of the GUI, some or all of such shown information may be user-selectable controls (or associated with such controls), which allow the end user to select and display some or all of the associated information (e.g., select the 360° panoramic image indicator for location 210B to view part or all of the panoramic image (e.g., in a manner similar to Figures 2A - 2C ). Additionally, in this example, user-selectable control 221 is added to indicate the current floor for the floor plan display and to allow the end user to select a different floor to display. In some embodiments, a change in floor or other level may also be made directly from the floor plan, such as via selection of the corresponding connecting passageway in the shown floor plan (e.g., stairs to floor 2), and other visual changes may be made directly from the shown floor plan by selecting the corresponding user-selectable control shown, (e.g., selecting the control corresponding to a particular image at a particular location and receiving the display of that image, instead of or in addition to the previous display of the floor plan of that image). In other embodiments, information for some or all of the different floors may be shown simultaneously, such as by showing separate sub floor plans for individual floors, or by integrating the room connection information for all rooms and floors into a single floor plan, the single floor Figure 1Shown together for the second time. It should be understood that in some embodiments, various other types of information may be added, in some embodiments some of the types of information shown may not be provided, and in other embodiments the visual indication and user selection of the linked and associated information may be displayed and selected in other ways.

[0052] Figure 2D Continuing with the example shown Figures 2A to 2C and showing additional information regarding the living room and regarding the analysis of 360° images taken in the living room (e.g., for a target panoramic image taken at two or more of acquisition locations 210A, 210B, and 210C) as part of a type of estimate of the likely shape of a portion of the room, such as through the MIGM system. Specifically, Figure 2D includes information 255de, which shows that the 360° image taken from location 240b in living room 201 will share information about various visible 2D features with the information of the 360° image taken from location 240a, although for simplicity, Figure 2D only a limited subset of such features of the portion of the living room is shown in Figure 2D Exemplary sight lines 228 from location 240b to various exemplary features in the room are shown, and similar exemplary sight lines 227 from location 240a to the corresponding features are shown, which show the degree of difference between the views at significantly separated capture locations. Figure 2DAlso shown is information 255df about the northeast portion of the living room visible in a subset of the 360° images taken from positions 240a and 240b (e.g., video frames taken along the sequence of position 240), and information 255dg about the northwest portion of the living room visible in other subsets of the 360° images taken from positions 240a and 240b, where various example features in those portions of the living room are visible in two 360° image frames (e.g., corners 195-1 and 195-2, windows 196-1 and 196-2, etc.). As part of the automatic analysis of the 360° images, partial information about planes 286e and 286f corresponding to portions of the north wall of the living room can be determined based on the detected features, and information 287e and 285f about portions of the east and west walls of the living room can be similarly determined based on the corresponding features identified in the images. In addition to identifying such partial plane information for the detected features (e.g., for each point in the determined sparse 3D point cloud from the image analysis), this automatic technique (optionally including SLAM and / or MVS and / or SFM techniques) can also determine information about the possible positions and orientations / directions 220 of the image subset from capture position 240a and the possible positions and orientations / directions 222 of the image subset from capture position 240b (e.g., the positions 220g and 222g of capture positions 240a and 240b, and the optional orientations 220e and 222e of the image subsets shown in 255df; and the corresponding positions 220g and 222g of capture positions 240a and 240b, and the optional orientations 220f and 222f for the image subsets shown in 255dg). Although only the features of portions of the living room are shown in Figure 2D , it should be understood that other portions of the 360° images corresponding to other portions of the living room can be analyzed in a similar manner to determine possible information about the possible planes of the various walls of the room and other features (not shown) in the living room. Similar analysis can be performed between some or all of the other images at position 215 in the living room that are selected for use, resulting in various determined feature planes from the various image analyses that can correspond to portions of the walls of the room.

[0053] Figure 2DFurther shown is information 255dh regarding a plurality of feature planes determined by analyzing 360° images captured at positions 240a and 240b, which feature planes may correspond to portions of the west wall and the north wall of the living room. The shown plane information includes a determined plane 286g near or at the north wall (and thus the possible position of the corresponding portion of the north wall), and a determined plane 285g near or at the west wall (and thus the possible position of the corresponding portion of the west wall). As expected, there are many variations in the different determined planes for the north wall and the west wall from the different features detected in the analysis of the two 360° images, such as differences in position, angle, and / or length, as well as missing data for some portions of the walls, resulting in uncertainty about the actual exact position and angle of each wall. Although not shown in Figure 2D , it will be understood that similar determined feature planes for the other walls of the living room will be detected similarly, as well as determined feature planes corresponding to features not along the walls (e.g., furniture). Figure 2D Also shown is information 255Di regarding additional determined feature plane information, which may correspond to portions of the west wall and the north wall of the living room. From the analysis of various additional 360° images in the living room, it is expected that in this example, the analysis of the additional images provides even greater variations in the different determined planes for the north wall and the west wall. Figure 2D Also shown is additional determined information for aggregating information about various portions of the determined feature planes in order to identify possible partial positions 295a and 295b of the west wall and the north wall, as shown in Figure 2D the information 255dj. Specifically, Figure 2D shown is information 291a regarding the normal orthogonal direction of some of the determined feature planes corresponding to the west wall, and additional information 288a regarding those determined feature planes. In an exemplary embodiment, the determined feature planes are clustered to represent the hypothesized wall position of the west wall, and the information about the hypothesized wall position is combined to determine a possible wall position 295a, such as by weighting information from various clusters and / or the determined feature planes below. In at least some embodiments, machine learning techniques are used to analyze the hypothesized wall position and / or the normal information to determine the resulting possible wall position, optionally by further applying assumptions or other constraints (e.g., Figure 2D the 90° angle shown in the information 289, and / or having a flat wall), as part of the machine learning analysis or the analysis result. A similar analysis may be performed on the north wall using the information 288b regarding the corresponding determined feature planes and the additional information 291b regarding the resulting normal orthogonal direction for at least some of those determined feature planes.Figure 2D Possible partial wall positions 295a and 295b for the west wall and the north wall of the living room are shown respectively.

[0054] Although not shown in Figure 2D , it will be understood that similarly, the similar determined feature planes and corresponding normal directions for other walls of the living room will be detected and analyzed to determine their possible positions, thereby obtaining an estimated partial overall room shape of the living room, which is based on visual data acquired by one or more image acquisition devices in the living room. Additionally, similar analysis is performed for each room of the building, thereby providing an estimated partial room shape for each room. Although not shown in Figure 2D , in at least some embodiments, only a single target panoramic image may be used to perform other room shape estimation operations, such as analyzing the visual data of the target panoramic image through one or more trained neural networks, as discussed in more detail elsewhere herein. Additionally, although not shown in Figure 2D either, in some embodiments, the analysis of the visual data captured by one or more image acquisition devices in the living room may be supplemented and / or replaced by analyzing depth data (not shown) captured by one or more image acquisition devices in the living room, such as directly generating an estimated 3D point cloud from depth data representing the walls of the living room and optionally the ceiling and / or floor, as Figure 2E shown. In particular, Figure 2E continues to show the example of Figures 2A to 2D and shows information 255E regarding additional information that can be generated from one or more images in a room and used in one or more ways. In particular, images (e.g., video frames) captured in the living room of house 198 may be analyzed to determine an estimated 3D shape of the living room, such as from a 3D point cloud of features detected in the video frames (e.g., using SLAM and / or SFM and / or MVS techniques, and optionally further based on IMU data captured by one or more image acquisition devices). In this example, information 255e is similar to Figure 2Bin a manner of the image 250c, reflecting an example portion of such a point cloud for the living room, such as corresponding to the northwest portion of the living room in this example (e.g., including the northwest corner 195-1 of the living room and the window 196-1). Such a point cloud can be further analyzed to detect features such as windows, doorways, and other room-to-room openings. In this example, the area 299 corresponding to the window 196-1 and the boundary 298 corresponding to the north wall of the living room are identified. It should be understood that in other embodiments, such an estimated 3D shape of the living room can be determined by using depth data captured by one or more image acquisition devices in the living room, regardless of whether visual data of one or more images captured by one or more image acquisition devices in the living room is used, or in addition to using visual data of one or more images captured by one or more image acquisition devices in the living room. Additionally, it should be understood that in the living room and other rooms of the house 198, various other walls and other features can be similarly identified.

[0055] Figure 2F Continuing to show Figures 2A to 2E an example, and showing additional information 255f, the additional information 255f corresponds to after determining the final estimated room shape for the rooms on the shown floor of the house 198 (e.g., the 2D room shape 236 for the living room), at least partially based on connecting room passages between rooms and matching room shape information between adjacent rooms, positioning the estimated room shapes of the rooms relative to each other. In at least some embodiments, such information can be regarded as constraints on room positioning and determining an optimal or otherwise preferred solution for these constraints. Figure 2F Examples of such constraints in include matching 231 the connection passage information of adjacent rooms (e.g., the passages detected in the automatic image analysis discussed in Figure 2D such that the positions of these passages are co-located, and matching 232 the shapes of adjacent rooms in order to connect these shapes (e.g., as shown for rooms 229d and 229e and for rooms 229a and 229b). In addition to or as an alternative to constraints based on passages and / or based on room shapes, in other embodiments, various other types of information can be used for the room shape positions, such as precise or approximate dimensions for the overall size of the house (e.g., based on additional available metadata about the building, analysis of images from one or more image acquisition locations outside the building, etc.). External information 233 outside the house can also be identified and used as a constraint (e.g., at least partially based on the automatic identification of passages and other features such as windows corresponding to the outside of the building) to prevent another room from being placed in a position that has been identified as the outside of the building. In Figure 2FIn the example, the finally estimated room shape used may be a 2D room shape, or alternatively, a 2D version of the 3D finally estimated room shape may be generated and used (e.g., by obtaining a horizontal slice of the 3D room shape).

[0056] Figure 2G and Figure 2H Continuing to show Figures 2A to 2F the example, and showing the mapping information that can be generated from the type of analysis discussed in Figures 2D to 2F . Specifically, Figure 2G shows an exemplary floor plan 230g, which can be constructed based on the positioning of the finally estimated room shape determined, in this example, the finally estimated room shape includes walls and indications of doorways and windows, and before determining and adding other building information as shown in the floor plan 230c of Figure 2C . Figure 2H Shows additional information 265h that can be generated and displayed (e.g., in a GUI similar to Figure 2C ) from the automatic analysis techniques disclosed herein. The additional information 265h in this instance is a 2.5D or 3D model floor plan of a house. Such a model 265h can be additional drawing-related information generated based on the floor plan 230g, which shows additional information about height in order to show the visible positions of features such as windows and doors in the walls, or alternatively by combining the finally estimated room shape of the 3D shape. Although not shown in Figure 2H , various types of additional information can be determined for a building and added to the floor plan model 265h in some embodiments and situations, such as the type of information shown by the floor position 230c of Figure 2C . Additionally, in some embodiments, further information can be added to the displayed walls, such as from images taken during video shooting (e.g., to present and show the actual paint, wallpaper, or other surfaces of the house from the presented model 265), and / or can otherwise be used to add specified colors, textures, or other visual information to the walls and / or other surfaces.

[0057] Figure 2I Continuing to show Figures 2A to 2H the example, and showing the information 255i, which shows an example of identifying defined types of objects in an image and using the standardized or expected sizes of these types of objects to determine the estimated camera height and other scaling information of the image. In this example, the image is from Figure 2Ca panoramic image 255c in an equal-rectangular format, and the defined object type is a doorway, where the swinging doorway 190-1 and the sliding doorway 190-6 are visible in the living room. The BOBSDM system analyzes the visual data of the image to identify the two doorways, and also determines additional information about the doorways, including identifying the four corners labeled A, B, C, and D of the doorway 190-1 and the bottom 264b-1 and the top 264a-11, and similarly identifying the top and bottom 264a-12 and 264b-12 and the four corners (not shown) of the doorway 190-6. In the illustrated embodiment, the BOBSDM system uses a hypothesized doorway height 254-11 of 80 inches (or 2.03 m) for the doorway 190-1 and desires to determine the portion of the height H1 corresponding to the estimated camera height above the floor when the image is taken, where the additional height H2 is the difference between the 80-inch height and the estimated camera height. As Figure 2I shown in the information 252, the corners can initially be represented in the coordinates of the image local, such as representing the lower left corner A as (x1, 1, z1), and then it can be transformed into the real-world coordinates by using the shown formula to calculate the estimated camera height H1. For the doorway 190-6, the same expected height of 80 inches for the height 254-12 of this doorway can be similarly used to determine a different estimated camera height H1' (not shown), and in some embodiments, the two estimated camera heights H1 and H1' can be combined to determine a single final estimated camera height of the image 255c, so that if the different estimated camera heights meet one or more defined verification criteria, the average value or other average value of the two heights is determined. (For example, differing by at most a defined threshold amount), otherwise alternative techniques are used to determine the building dimension information, as Figure 2L discussed in. Additionally, although not shown in Figure 2I , other objects of other defined types visible in the same image (e.g., floor cabinets) can be similarly analyzed to determine other individual estimates of the camera height by using the standardized or otherwise expected heights of those other object types (e.g., 36 inches for floor cabinets), and if so, the final estimated camera height of the image 255c can be further based on those other additional individual estimated camera heights, and if the different estimated camera heights meet one or more defined verification criteria (e.g., differing by at most a defined threshold amount), although such other object types are not shown in this example. Additionally, although in Figure 2Iis not shown, but similar estimated camera heights may be determined for objects in other images taken during the same image acquisition session, such as other doorways and / or other objects of other defined types, and if so, the final estimated camera height for image 255c may also be based on those other additional individual estimated camera heights and if different estimated camera heights meet one or more defined validation criteria (e.g., differ by at most a defined threshold amount), although such other images and associated other objects are not shown in this example. Once the estimated camera height is determined, the actual distance may be determined and assigned to other objects and elements visible in the image, such as the width of doorway 190-1, the height and width / length 253-i1 of the picture window on the left wall of the living room, the width / length 253-i2 of the north wall of the living room, etc. By determining at least the width / length of each wall, the dimensions of the 2D room shape of the living room generated from the visual data of the image may be determined, and if the height information of one or more walls is further determined, the dimensions of the 3D room shape of the living room may be similarly determined.

[0058] Figure 2J and Figure 2K continues to show Figures 2A to 2I the example, and shows information 255J and 255K, which show additional examples of identifying objects of the defined type in other images. Specifically, referring to Figure 2J , several floor cabinets are shown, and the BOBSDM system has identified the tops 262a-j and bottoms 262b-j of these floor cabinets, additional information 261a-j representing the floor-to-wall boundary, and the bottoms 264b-j and tops 264a-j of the visible doorways. As discussed in more detail elsewhere herein and in a manner similar to Figure 2I , the BOBSDM system may use the information shown for the visible doorways to determine a first estimated camera height and may determine one or more second estimated camera heights for the visible floor cabinets, such as one estimated camera height for each cabinet, one estimated camera height for each pixel column in which the floor cabinet is visible, etc. As discussed with respect to Figure 2I and elsewhere in this document, the estimated camera heights of various visible objects of the defined type may then be combined to determine the final estimated camera height for at least the current image (and optionally, additional images taken during the same image acquisition session), including by further combining the estimated camera heights determined from one or more objects of the defined type visible in those other images). Figure 2KAnother example of identifying one or more objects of a defined type is provided. The example shown includes various floor cabinets and the identified doorways, and the BOBSDM system identifies the tops 262a-k and bottoms 262b-k of the floor cabinets, the floor-to-wall boundaries 261a-k, and the tops and bottoms 264a-k and 264b-k of the visible doorways.

[0059] Figure 2L Continuing with the example shown Figures 2A to 2K and information 255L is shown regarding alternative techniques for determining the dimensions of a building and its rooms, such as without determining an estimated camera height based on the analysis of one or more images (e.g., if the different individual estimated camera heights of multiple objects of one or more defined object types differ by more than a defined threshold amount, or do not meet one or more defined verification criteria). In the example shown, an aerial image 2501 of a building 198 has been obtained (e.g., from a drone or satellite), and a floor plan for the building has been generated (e.g., from analyzing visual data of images taken in and around the building). The automated techniques of the BOBSDM system include fitting at least a portion of the exterior of an entire building floor plan (e.g., a partial building floor plan for the first floor) to the visible exterior of the building in the image, such as by starting with the dimensions 256a of the floor plan and reducing them until a fit 256b is achieved. It should be understood that in other embodiments, the fitting can be performed in other ways, including starting with a visual representation of a floor plan that is smaller than the visible exterior of the building in the image and increasing its dimensions, and performing rotations and other operations if the image is not taken directly from above and is not aligned with the same orientation information used for the floor plan.

[0060] Various details have been provided Figures 2A to 2L but it should be understood that the details provided are non-exclusive examples included for illustrative purposes, and other embodiments can be performed in other ways without some or all of such details.

[0061] As described above, in some embodiments, the techniques described include using machine learning to learn the properties and / or other characteristics of an adjacency graph for encoding in a corresponding vector embedding generated, such as the properties and / or other characteristics of a building floor plan that are most likely to subsequently automatically identify a building floor plan having properties that meet a target criterion, and using the vector embedding, in at least some embodiments, to identify the target building floor plan being encoded based on such learned properties or other characteristics. In particular, in at least some such embodiments, graph representation learning is used to search for a mapping function that can map nodes in a graph to d-dimensional vectors such that similar nodes in the graph have similar embeddings in the learned space. Different from traditional methods such as graph kernel methods (e.g., see "Graph Kernels" by S.V.N. Vishwanathan et al., Journal of Machine Learning Research, 11: 1201-1242, 2010; and "A Survey On Graph Kernels" by Nils M. Kriege et al., arXiv:1903.11835, 2019), graph neural networks eliminate the process of hand-designing features and directly learn high-level embeddings from the raw features of nodes or (sub) graphs. There are various techniques for extending and redefining convolutions in the graph domain, which can be classified as spectral methods and spatial methods.Spectral methods adopt the spectral representation of graphs and are specialized for specific graph structures, making the models trained on one graph inapplicable to graphs with different structures (e.g., see "Spectral Networks And Locally Connected Networks On Graphs" by Joan Bruna et al., International Conference on Learning Representations 2014, 2014; "Convolutional Neural Networks On Graphs With Fast Localized Spectral Filtering" by Michael Defferrard et al., Proceedings of Neural Information Processing Systems 2016, 2016, pp. 3844 - 3852; and "Semi-Supervised Classification With Graph Convolutional Networks" by Thomas N. Kipf et al., International Conference on Learning Representations 2014, 2017). The convolutional operations of spectral methods are defined in the Fourier domain by computing the eigen-decomposition of the graph Laplacian, and the filters can be approximated by the Chebyshev expansion of the graph Laplacian to reduce the expensive eigen-decomposition, thereby generating local filters, where the filters are optionally restricted to work on adjacent nodes one step away from the current node. Regarding spatial methods, it involves learning the embeddings of nodes by recursively aggregating information from the local neighbors of the nodes. Various numbers of adjacent nodes and corresponding aggregation functions can be processed in various ways.For example, a fixed number of neighbors of each node can be sampled, and different aggregation functions can be used, such as mean, max, and long short-term memory network (LSTM) (e.g., see "Inductive Representation Learning On Large Graphs" by Will Hamilton et al., Proceedings of Neural Information Processing Systems 2017, 2017, pp. 1024–1034). Optionally, each adjacent node can be considered to contribute differently to the central node, and the contribution factor can be learned through a self-attention model (e.g., see "Graph Attention Networks" by P. Velickovic et al., International Conference on Learning Representations 2018, 2018). Additionally, each attention head captures feature correlations in different representation subspaces and can be processed differently, such as by using a convolutional subnetwork to weight the importance of each attention head (e.g., see "GaAN: Gated Attention Networks For Learning On Large And Spatiotemporal Graphs" by Jiani Zhang et al., Proceedings of Uncertainty in Artificial Intelligence 2018, 2018).

[0062] Additionally, in some embodiments, creating an adjacency graph and / or an associated vector embedding for a building may also be based in part on partial information provided for the building (e.g., by an operator user of the BOBSDM system, by one or more end users, etc.). Such partial information may include, for example, one or more of the following: some or all of the room names of the provided building rooms, where connections between the rooms will be automatically determined or otherwise established; providing some or all of the inter-room connections between the rooms of the building, with possible room names for automatically determining or establishing the rooms; providing some room names and inter-room connections, with other inter-room connections and / or possible room names being automatically determined or otherwise established. In such embodiments, the automated techniques may include using the partial information as part of completing or otherwise generating a floor plan of the building, where the floor plan is subsequently used to create the corresponding adjacency graph and / or vector embedding.

[0063] Figure 3 FIG. 4 is a block diagram of an embodiment showing one or more server computing systems 300 implementing the BOBSDM system 340 and one or more server computing systems 380 implementing the ICA system 388 and the MIGM system 389. The (multiple) server computing systems and the BOBSDM and / or ICA and / or MIGM systems may be implemented using a plurality of hardware components that form electronic circuitry adapted and configured to perform at least some of the techniques described herein when in combined operation. One or more computing systems and devices may also optionally execute a building information access system, not shown (e.g., the (multiple) server computing systems 300, client computing devices (multiple), etc.) and / or optional other programs 335 and 383 (e.g., the (multiple) server computing systems 300 and 380 in this example). In the illustrated embodiment, each server computing system 300 includes one or more hardware central processing units (“CPUs”) or other hardware processors 305, various input / output (“I / O”) components 310, a storage device 320, and a memory 330. The illustrated I / O components include a display 311, a network connection 312, a computer-readable media drive 313, and other I / O devices 315 (e.g., I / O devices, keyboard, mouse or other pointing device, microphone, speaker, GPS receiver, etc.). Each server computing system 380 may have similar components, although for simplicity only one or more hardware processors 381, a memory 387, a storage device 384, and I / O components 382 are shown in this example.

[0064] In the illustrated embodiment, the BOBSDM system 340 executes in the memory 330 of the (multiple) server computing systems 300 to perform at least some of the described techniques, such as by using the processor 305 to configure the processor 305 and the computing system 300 to execute the software instructions of the system 340 in a manner that implements the described techniques for automated operations. The illustrated embodiment of the BOBSDM system may include one or more components not shown to each perform a portion of the functions of the BOBSDM system, such as in the manner discussed elsewhere herein, and the memory may further optionally execute one or more other programs 335. As a specific example, in at least some embodiments, a copy of the ICA and / or MIGM system may be executed as one of the other programs 335, e.g., instead of or in addition to the ICA system 388 or the MIGM system 389 on the (multiple) server computing systems 380, and / or a copy of the building information access system may be executed as one of the other programs 335. The BOBSDM system 340 may also store and / or retrieve various types of data (e.g., in one or more databases or other data structures) during its operation on the memory 320, such as information about the identified target objects (e.g., doorways and floor cabinets) and information 326 from the building images, the determined camera height estimates and other image scaling information 325, the determined building dimension information 327 (e.g., dimensions of target objects and other visible objects / elements, walls, rooms, the building as a whole, etc.), floor plans and images and other associated information 324 (e.g., images captured and / or generated by the ICA system; 2D and / or 2.5D and / or 3D models generated by the MIGM system; building and room dimensions used in conjunction with, such as generated by, the BOBSDM system; additional images and / or annotation information, etc.), optionally various types of user information 322 for users who interact with the BOBSDM system, and / or various types of optional additional information 329 (e.g., various analysis information related to the presentation or other use of one or more building interiors or other environments). Figure 1 Starting with the building and room dimensions used in conjunction with, such as generated by, the BOBSDM system; additional images and / or annotation information, etc.)

[0065] Additionally, the implementations of the ICA system 388 and the MIGM system 389 in the illustrated embodiments are executed in the memory 387 of the server computing system 380 to perform techniques related to generating panoramic images and floor plans of a building, such as by using the processor 381 to configure the processor 381 and the computing system 380 to execute software instructions of the systems 388 and / or 389 in a manner that performs automated operations implementing these techniques. The illustrated implementations of the ICA and MIGM systems may include one or more components not shown to perform portions of the functions of the ICA and MIGM systems, respectively, and the memory may further optionally execute one or more other programs 383. The ICA system 388 or the MIGM system 389 may also store and / or retrieve various types of data during operation on the storage device 384 (e.g., in one or more databases or other data structures), such as video and / or image information 386 obtained for one or more buildings (e.g., for analysis to generate a floor plan, for providing a 360° video or image for display to a user of the client computing device 390, etc.), floor plans and / or other generated mapping information 387, and optionally other information 385 (e.g., additional image and / or annotation information for use with an associated floor Figure 1 for use with an associated floor Figure 1 dimensions of buildings and rooms for use with an associated floor, various analysis information related to the presentation or other use of one or more building interiors or other environments, etc.). Although not shown in Figure 3 the ICA and / or MIGM systems may also store and use additional types of information, such as other types of building information to be analyzed and / or provided to the BOBSDM system, information about ICA and / or MIGM system operator users and / or end users, etc.

[0066] One or more server computing systems 300 and executing the BOBSDM system 340, (multiple) server computing systems 380 and executing the ICA system 388 and the MIGM system 389, and optionally executing a building information access system (not shown) can communicate with each other and with other computing systems and devices, such as via one or more networks 399 (e.g., the Internet, one or more cellular phone networks, etc.), including interacting with user client computing devices 390 (e.g., for viewing floor plans, and optionally associated images and / or other relevant information, such as by interacting with or executing a copy of the building information access system), and / or mobile image acquisition devices 360 (e.g., for acquiring images and / or other information of a building or other environment to be modeled), and / or optionally receiving and using other navigable devices 395 for navigation purposes with floor plans and optionally other generated information (e.g., used by semi - autonomous or fully autonomous vehicles or other devices). In other embodiments, some of the described functionality can be combined in fewer computing systems, so as to combine the BOBSDM system 340 and the building information access system in a single system or device, combine the image acquisition functionality of the BOBSDM system 340 and the device 360 in a single system or device, combine the ICA system 388 and the MIGM system 389 and the image acquisition functionality of the device 360 in a single system or device, combine the BOBSDM system 340 and the ICA system 388 and the MIGM system 389 in a single system or device, combine the BOBSDM system 340 and the ICA system 388 and the MIGM system 389 and the image acquisition functionality of the device 360 in a single system or device, etc.

[0067] Some or all of the user client computing devices 390 (e.g., mobile devices), mobile image acquisition devices 360, optional other navigable devices 395, and other computing systems (not shown) may similarly include some or all of the same types of components shown for the server computing system 300. As a non-limiting example, each of the mobile image acquisition devices 360 is shown as including one or more hardware CPUs 361, I / O components 362, memory and / or storage devices 367, one or more imaging systems 365, IMU hardware sensors 369 (e.g., for acquiring video and / or images, associated device movement data, etc.), and optional other components 364. In the example shown, one or both of a browser and one or more client applications 368 (e.g., applications dedicated to the BOBSDM system and / or ICA system and / or MIGM system) are executed in the memory 367 to participate in communication with the BOBSDM system 340, ICA system 388, MIGM system 389, and / or other computing systems. Although specific components are not described for the other navigable devices 395 or other computing devices / systems 390, it will be understood that they may include similar and / or additional components.

[0068] It should also be understood that Figure 3 the computing systems 300 and 380 and other systems and devices included in are merely illustrative and not intended to limit the scope of the present invention. The systems and / or devices may instead each include multiple interactive computing systems or devices and may be connected to other devices not specifically shown, including via Bluetooth communication or other direct communication, via one or more networks such as the Internet, via the Web, or via one or more private networks (e.g., mobile communication networks, etc.). More generally, the devices or other computing systems may include any combination of hardware that can interact and perform the types of functions, optionally when programmed or otherwise configured with specific software instructions and / or data structures, including but not limited to desktop computers or other computers (e.g., input tablets, tablet computers, etc.), database servers, network storage devices and other network devices, smart phones and other cellular phones, consumer electronic devices, wearable devices, digital music player devices, handheld gaming devices, PDAs, wireless telephones, Internet devices, and various other consumer products including appropriate communication capabilities. Additionally, in some embodiments, the functions provided by the illustrated BOBSDM system 340 may be distributed among various components, some of the functions of the BOBSDM system 340 may not be provided, and / or other additional functions may be provided.

[0069] It should also be understood that although various items are shown as being stored in memory or in a storage device when in use, for purposes of memory management and data integrity, these items or portions thereof may be transferred between memory and other storage devices. Optionally, in other embodiments, some or all of the software components and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Thus, in some embodiments, when configured by one or more software programs (e.g., the BOBSDM system 340 executed on the server computing system 300, the building information access system executed on the server computing system 300 or other computing systems / devices, etc.) and / or data structures, some or all of the described techniques may be executed by a hardware device including one or more processors and / or memory and / or storage devices. Such as by executing software instructions of one or more software programs and / or by storing such software instructions and / or data structures and in order to execute algorithms as described in the flowcharts and other disclosures herein. Additionally, in some embodiments, some or all of the systems and / or components may be implemented or provided in other ways, such as by consisting of one or more devices (e.g., devices not implemented in whole or in part by software instructions configuring a particular CPU or other processor) that are implemented in part or in whole in firmware and / or hardware, including but not limited to one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device that will be read by an appropriate drive or via an appropriate connection, volatile or non-volatile memory (e.g., RAM or flash RAM), network storage devices, or portable media articles (e.g., DVD disks, CD disks, optical disks, flash devices, etc.). In some embodiments, the systems, components, and data structures may also be transmitted on various computer-readable transmission media via generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal), the computer-readable transmission media including wireless and wire / cable-based media and may take various forms (e.g., as part of a single or multiplexed analog signal or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Thus, the embodiments of the present disclosure may be implemented with other computer system configurations.

[0070] Figure 4FIG. 0 shows an example flow chart of an embodiment of an ICA (Image Capture and Analysis) system routine 400. The routine may be performed by, for example, Figure 1 ICA system 160 of Figure 3 , ICA system 388 of Figures 2A - 2C , and / or an ICA system as described with respect to Figures 5A - 5B and elsewhere herein to obtain 360° panoramic images and / or other images at an acquisition location within a building or other structure, for example, for subsequent generation of related floor plans and / or other mapping information. Although portions of example routine 400 are discussed with respect to obtaining a particular type of image at a particular acquisition location, it will be understood that the routine or a similar routine may be used to obtain video (with video frame images) and / or other data (e.g., audio), either in lieu of such panoramic images or other stereoscopic images, or in addition to such panoramic images or other stereoscopic images. Additionally, while the illustrated embodiment obtains and uses information from within a target building, it should be understood that other embodiments may perform similar techniques on other types of data, including information external to non-building structures and / or to one or more target buildings of interest. Further, some or all of the routine may be performed on a mobile device used by a user to obtain image information, and / or some or all of the routine may be performed by a system remote from such a mobile device. In at least some embodiments, routine 400 may be called from block 512 of Figures 5A - 5B routine 500, and the corresponding information from routine 400 may be provided to routine 500 as part of the implementation of that block 512. In other embodiments, routine 500 may perform additional operations asynchronously without waiting to return such processing control (e.g., performing other processing activities while waiting for the corresponding information from routine 400 to be provided to routine 500 at block 512).

[0071] The illustrated embodiment of the routine begins at block 405, where instructions or information are received. At block 410, the routine determines whether the received instructions or information indicate obtaining visual data and / or other data representative of the interior of a building (optionally obtaining instructions based on the provided information about one or more additional obtaining locations and / or other guidance), and if not, proceeds to block 490. Otherwise, the routine proceeds to block 412 to receive an indication to begin an image obtaining process at a first obtaining location (e.g., from a user of a mobile image obtaining device that will perform the obtaining process). After block 412, the routine proceeds to block 415 to perform an obtaining location image obtaining activity, the obtaining location image obtaining activity being for obtaining, via one or more fisheye lenses and / or non-fisheye rectilinear lenses on a mobile device, a 360° panoramic image of the obtaining location of a target building interior of interest and providing at least 360° of horizontal coverage about a vertical axis, although other types of images and / or other types of data may be obtained in other embodiments. As a non-exclusive example, the mobile image obtaining device may be a rotating (scanning) panoramic camera equipped with a fisheye lens (e.g., having a 180° horizontal coverage) and / or other lenses (e.g., having a horizontal coverage less than 180°, such as a conventional lens or a wide-angle lens or an ultra-wide lens). The routine may also optionally obtain from the user annotations and / or other information about the obtaining location and / or the surrounding environment, e.g., for later use when presenting information about the obtaining location and / or the surrounding environment.

[0072] After the completion of block 415, the routine proceeds to block 425 to determine whether there are more acquisition locations for acquiring images, e.g., based on information provided by the user of the mobile device and / or received at block 405. In some embodiments, the ICA routine will acquire only a single image and then proceed to perform blocks 430 and 4407 to provide the image and corresponding information (e.g., return the image and corresponding information to the MIGM system for further use before receiving additional instructions or information to acquire one or more next images at one or more next acquisition locations). If there are more acquisition locations at the current time to acquire additional images, the routine proceeds to block 427 to optionally initiate the capture of link information while the mobile device is moving along a travel path away from the current acquisition location and towards the next acquisition location within the building interior. The captured link information may include additional sensor data (e.g., from one or more IMUs or inertial measurement units, on the mobile device or otherwise carried by the user) and / or additional visual information (e.g., images, videos, etc.) recorded during such movement. The initiation of the capture of such link information may be performed in response to an explicit indication from the user of the mobile device or based on one or more automated analyses of information recorded from the mobile device. Additionally, in some embodiments, during the movement to the next acquisition location, the routine may also optionally monitor the movement of the mobile device and provide one or more guiding prompts (e.g., to the user) regarding the movement of the mobile device, the quality of the captured sensor data and / or visual information, the associated lighting / environmental conditions, the desirability of capturing the next acquisition location, and any other appropriate aspects of capturing the link information. Similarly, the routine may optionally obtain annotations and / or other information from the user regarding the travel path, e.g., for use later in connecting links between panoramic images presenting information or results regarding the travel path. At block 429, the routine determines that the mobile device has reached the next acquisition location (e.g., based on an indication from the user, based on the forward movement of the mobile device stopping for at least a predetermined amount of time, etc.), uses it as the new current acquisition location, and returns to block 415 to perform the acquisition location image acquisition activity for the new current acquisition location.

[0073] If it is determined in block 425 that there are no more acquisition locations for obtaining image information of the current building or other structure at the current time, the routine proceeds to block 430 to optionally analyze information about one or more acquisition locations to identify possible additional areas in the building for obtaining visual data (e.g., based on not obtaining visual data of the kitchen or bathroom, obtaining visual data of only 2 bathrooms while the textual description information of the building indicates 3 bathrooms, etc.) and / or other information to collect (e.g., audio data), and if so identified, optionally further provide user suggestions and / or instructions and / or otherwise assist in capturing the corresponding additional data. In block 435, the routine then optionally preprocesses the acquired 360° panoramic image before the acquired 360° panoramic image is subsequently used (e.g., for generating relevant mapping information, for providing information about the characteristics of a room or other enclosed area, etc.) in order to produce an image of a specific type and / or specific format (e.g., generating a rectilinear equirectangular projection for each such image, which has straight vertical data, such as the sides of a typical rectangular doorframe or the straight boundary between 2 adjacent walls, and has straight horizontal data, such as the top of a typical rectangular doorframe or the boundary between the wall and the floor that remains straight at the horizontal midline of the image, but gradually curves in a manner that bulges relative to the horizontal midline in the equirectangular projection image as the distance from the horizontal midline in the image increases). After block 435, the routine continues to block 440 to execute the Building Object Scale Determination Manager (BOBSDM) routine to determine the estimated camera height of the image acquired in block 415 and optional other scaling information, and use this scaling information to determine building dimension information, where an example of such a routine further relative to Figure 6 is shown. After receiving the result of the BOBSM routine in block 440, in block 480 the image and any associated generated or obtained information are stored for later use and are optionally provided to one or more recipients (e.g., provided to block 512 of routine 500 if called from this block). Figures 5A to 5B An example of a routine for generating a floor plan representation of the interior of a building from the generated panoramic information is shown.

[0074] If it is determined in block 410 that the instruction or other information received in block 405 is not to acquire images and other data representing the interior of a building, the routine proceeds to block 490 to appropriately perform any other indicated operations to configure parameters to be used in various operations of the system (e.g., at least partially based on information specified by a user of the system (e.g., a user of a mobile device capturing one or more interiors of a building, an operator user of an ICA system, etc.), in response to a request to generate and store information (e.g., identifying a set or sets of interconnected linked panoramic images, each panoramic image representing a building or a portion of a building that matches one or more specified search criteria, one or more panoramic images that match one or more specified search criteria, etc.), to generate and store inter-panoramic connections between panoramic images of a building or other structure (e.g., for each panoramic image, determining the direction within that panoramic image toward one or more other acquisition locations of one or more other panoramic images such that arrows or other visual representations with the panoramic images can be later displayed, and for each such determined direction from a panoramic image, enabling an end user to select one of the displayed visual representations to switch to the display of another panoramic image at the other acquisition location corresponding to the selected visual representation), to obtain and store other information about the user of the system, to perform any housekeeping tasks, etc.).

[0075] After block 480 or block 490, the routine proceeds to block 495 to determine whether to continue, e.g., until a clear indication to terminate is received, or to proceed only when a clear indication to continue is received. If it is determined to continue, the routine returns to block 405 to wait for additional instructions or information, and if not, proceeds to step 499 and ends.

[0076] Figures 5A to 5B An exemplary embodiment of a flowchart of a MIGM (Mapping Information Generation Manager) system routine 500 is shown. The routine can be executed by, for example, the MIGM system 160 of Figure 1 (not provided in the original, assumed to be a reference), the MIGM system 389 of Figure 3 (not provided in the original, assumed to be a reference), and / or a MIGM system as described with respect to Figures 2A to 2H (not provided in the original, assumed to be a reference) and elsewhere in this document, to determine the room shape (e.g., one or more 360° panoramic images) of a room (or other defined area) by analyzing information from one or more images acquired in the room, to generate a partial or complete floor plan of a building or other defined area at least partially based on one or more images of the area and optionally additional data captured by a mobile computing device, and / or to generate other mapping information of a building or other defined area at least partially based on one or more images of the area and optionally additional data captured by a mobile computing device. In Figures 5A to 5BIn the example, the determined room shape of the room can be a 2D room shape representing the locations of the walls of the room or a 3D fully closed combination of planar surfaces representing the locations of the walls, ceiling, and floor of the room, and the mapping information of the generated building (e.g., a house) can include a 2D floor plan and / or a 3D computer model floor plan. However, in other embodiments, other types of room shapes and / or mapping information can be generated and used in other ways, including for other types of structures and defined areas, as discussed elsewhere herein.

[0077] The illustrated embodiment of the routine begins at block 505, where information or instructions are received. The routine continues to block 510 to determine whether image information is already available for analyzing one or more rooms (e.g., for some or all of the indicated building, such as based on one or more such images previously generated by the ICA routine as received in block 505), or whether such image information is currently to be acquired. If it is determined in block 510 that some or all of the image information is currently to be acquired, the routine continues to block 512 to acquire such information, optionally waiting for one or more users or devices to move through one or more rooms of the building and acquiring panoramic images or other images at one or more acquisition locations in one or more of the rooms (e.g., multiple acquisition locations in each room of the building), optionally along with metadata information about the acquisition and / or interconnect information related to the movement between the acquisition locations, as discussed in more detail elsewhere herein. The implementation of block 512 can, for example, include invoking an ICA system routine to perform such activities, where Figure 4 An exemplary embodiment of an ICA system routine for performing such image acquisition is provided. If it is determined in block 510 that images are not currently being acquired, the routine continues to block 515 to obtain one or more existing panoramic images or other images from one or more acquisition locations in one or more of the rooms (e.g., multiple images acquired at multiple acquisition locations including at least one image and acquisition location in each room of the building), optionally along with metadata information about the acquisition and / or interconnect information related to the movement between the acquisition locations, e.g., which in some cases has been provided along with the corresponding instructions in block 505.

[0078] After block 512 or 515, the routine proceeds to block 520, where it determines whether to generate mapping information (sometimes referred to as a "virtual tour") that includes a set of links to target panoramic images (or other images) for a building or other group of rooms, such that an end user can move from any one of the images in the set of links to one or more other images linked to the starting current image, in some embodiments, including by selecting user-selectable controls for each such other linked image displayed with the current image, optionally by overlapping a visual representation of such user-selectable controls and the corresponding inter-image directions on the visual data of the current image, and similarly moving from the next image to one or more additional images linked to that next image, and so on), and if so, proceeds to block 525. The routine in block 525 selects at least some pairs of images (e.g., based on images that have overlapping visual content), and based on the shared visual content and / or based on other captured link interconnect information associated with the pair of images (e.g., movement information) (either moving directly from the acquisition location of one image in a pair to the acquisition location of the other image in the pair, or moving between these starting and ending acquisition locations via one or more other intermediate acquisition locations of other images). The routine in block 525 may further optionally use relative orientation information of at least the paired images to determine the global relative positions of some or all of the images with respect to each other in a common coordinate system, and / or generate inter-image links and corresponding user-selectable controls as described above. Additional details regarding creating such a set of links to images are included elsewhere in this document.

[0079] After block 525, or if it is determined in block 520 that the instruction or other information received in block 505 is not to determine a set of links to images, the routine proceeds to block 535 to determine whether the instruction received in block 505 indicates to generate other mapping information (e.g., a floor plan) for the indicated building, and if so, the routine proceeds to execute some or all of blocks 537 - 585 to do so, and otherwise proceeds to block 590. In block 537, the routine optionally obtains additional information about the building, such as from activities performed during the acquisition and optionally analysis of the images, and / or from one or more external sources (e.g., an online database, information provided by one or more end users, etc.). Such additional information may include, for example, the external dimensions and / or shape of the building, additional images and / or annotation information corresponding to specific locations outside the building (e.g., around the building and / or for other structures on the same property, from one or more overhead locations, etc.), additional images and / or annotation information corresponding to specific locations inside the building (optionally, for locations different from the acquisition locations of the captured panoramic images or other images), etc.

[0080] After block 537, the routine proceeds to block 540 to select the next room (starting with the first) for which one or more images (e.g., 360° panoramic images) acquired in the room are available, and analyze the visual data of the images for the room to determine the room shape (e.g., by determining at least wall locations), optionally along with determining uncertainty information regarding the walls and / or other parts of the room shape, and optionally including identifying other walls and floor and ceiling elements (e.g., wall structure elements / features such as windows, doorways, and stairs, as well as other inter-room wall openings and connecting passages, wall boundaries between a wall and another wall and / or ceiling and / or floor, etc.) and their positions within the determined room shape of the room. In some embodiments, room shape determination may include using the boundaries of the walls with each other and at least one of the floor or ceiling to determine a 2D room shape (e.g., using one or trained machine learning models), while in other embodiments, room shape determination may be performed in other ways (e.g., by generating a 3D point cloud of some or all of the room walls and optionally the ceiling and / or floor. For example, by analyzing at least the visual data of the panoramic images and optionally analyzing additional data captured by the image acquisition device or an associated mobile computing device, optionally using one or more of SFM (Structure from Motion), SLAM (Simultaneous Localization and Mapping), or MVS (Multi-View Stereo) analysis. Additionally, the activity at block 545 may optionally determine and use the initial pose information for each of those panoramic images (e.g., provided with acquisition metadata for the panoramic images), and / or obtain and use additional metadata for each panoramic image (e.g., acquisition height information of the camera device or other image acquisition device for acquiring the panoramic image relative to the floor and / or ceiling). Additional details regarding determining room shape and identifying additional information about the room are included elsewhere in this document. After block 540, the routine proceeds to block 545, where it determines whether there are more rooms for which to determine the room shape based on the images acquired in those rooms, and if so, returns to block 540 to select the next such room for which to determine the room shape.

[0081] If it is determined in block 545 that there are no more rooms for which to generate room shapes, the routine proceeds to block 560 to determine whether to further generate at least a partial floor plan of the building (e.g., at least in part based on the determined room shapes from block 540 and optionally further information regarding how the determined room shapes are to be positioned relative to each other). If not, e.g., when only one or more room shapes are determined without generating further mapping information of the building (e.g., determining the room shape of a single room based on one or more images acquired by the ICA system in the room), the routine proceeds to block 588. Otherwise, the routine proceeds to block 565 to retrieve one or more room shapes (e.g., the room shapes generated in block 545), or otherwise obtain one or more room shapes of the rooms of the building (e.g., based on manually provided input), whether 2D or 3D room shapes, and then proceeds to block 570. In block 570, the routine uses the one or more room shapes to create an initial floor plan (e.g., an initial 2D floor plan using 2D room shapes and / or an initial 3D floor plan using 3D room shapes), such as a partial floor plan that includes one or more but less than all of the room shapes of the building, or a complete floor plan that includes all of the room shapes of the building. If there are multiple room shapes, the routine further determines in block 570 the positioning of the room shapes relative to each other, e.g., by using visual overlaps between images from multiple acquisition locations to determine the relative positions of those acquisition locations and the room shapes around those acquisition locations, and / or by using other types of information (e.g., using inter-room corridors connecting rooms, optionally applying one or more constraints or optimizations, etc.). In at least some embodiments, the routine in block 570 further refines some or all of the room shapes by generating a binary segmentation mask that covers the relatively positioned room shapes, extracting polygons representing the outline or contour of the segmentation mask, and separating the polygons into refined room shapes. Such a floor plan may include, for example, relative position and shape information for various rooms without providing any actual dimension information for individual rooms or the building as a whole, and may also include multiple linked or associated subgraphs of the building (e.g., to reflect different floors, levels, sections, etc.). The routine also optionally associates the positions of doors, wall openings, and other identified wall elements on the floor plan.

[0082] After block 570, the routine optionally performs one or more of steps 580 through 585 to determine additional information and associate the additional information with the floor plan. In block 580, the routine optionally estimates the dimensions of some or all of the rooms, such as by analyzing the images and / or their acquisition metadata or from overall dimension information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan. It should be understood that if sufficiently detailed dimension information is available, a building drawing, blueprint, etc. can be generated from the floor plan. After block 580, the routine proceeds to block 583 to optionally associate further information with the floor plan (e.g., with a specific room or other location within the building), such as additional existing images with specified positions and / or annotation information. In block 585, if the room shape from block 545 is not a 3D room shape, the routine also optionally estimates the height of the walls in some or all of the rooms, such as based on the analysis of the images and the optionally known size of objects in the images, as well as information about the height of the camera when the images were acquired, and uses this height information to generate a 3D room shape for the room. The routine further optionally uses the 3D room shape (whether from block 540 or from block 585) to generate a 3D computer model floor plan of the building, where the 2D and 3D floor plans are associated with each other. In other embodiments, only the 3D computer model floor plan can be generated and used (including, if needed, by providing a visual representation of the 2D floor plan by using a horizontal slice of the 3D computer model floor plan).

[0083] After block 585, or if it is determined in block 560 that a floor plan is not being determined, the routine proceeds to block 588 to store the determined room shape and / or the generated mapping information and / or other generated information, optionally provide some or all of the information to one or more recipients (e.g., to block 440 of routine 400 if called from this block), and optionally further use some or all of the determined and generated information in order to provide the generated 2D floor plan and / or 3D computer model floor plan for display on one or more client devices and / or for display to one or more other devices, for use in automating the navigation of these devices and / or associated vehicles or other entities, and to similarly provide and use information about a set of links to the determined room shape and / or panoramic images and / or additional information about the determined room contents and / or passageways between rooms, etc.

[0084] If it is determined in block 535 that the information or instruction received in block 505 is not to generate mapping information for the indicated building, the routine proceeds to block 590 to appropriately perform one or more other indicated operations. Such other operations may include, for example, receiving and responding to requests for a previously generated floor plan and / or previously determined room shapes and / or other generated information (e.g., requests for such information to be displayed on one or more client devices, requests for such information to be provided to one or more other devices for use in autonomous navigation, etc.), obtaining and storing information about the building to be used in subsequent operations (e.g., information about the dimensions, quantity, or type of rooms, total square footage, other buildings in the vicinity or nearby, vegetation in the vicinity or nearby, external images, etc.), and so on.

[0085] After block 588 or block 590, the routine proceeds to block 595 to determine whether to continue, for example until a clear indication to terminate is received, or to continue only upon receiving a clear indication to continue. If it is determined to continue, the routine returns to block 505 to wait and receive additional instructions or information, and otherwise proceeds to block 599 and ends.

[0086] Although not directed at Figures 5A to 5BThe automated operations shown in the exemplary embodiments are described, but in some embodiments, a human user may further assist in facilitating some operations of the MIGM system, such as providing an operator user and / or an end user of the MIGM system with inputs of one or more types, which are further used in subsequent automated operations. As a non-exclusive example, such a human user may provide one or more types of inputs as follows: providing an input to assist in linking a collection of images so as to provide an input in box 525, which is used as part of the automated operation of that box (e.g., specifying or adjusting an initially automatically determined orientation between one or more pairs of images, specifying or adjusting an initially automatically determined final global position of some or all of the images relative to each other, etc.); providing an input in box 537, which is used as part of a subsequent automated operation, such as information of one or more of the types shown with respect to a building; providing an input with respect to box 540, which is used as part of a subsequent automated operation, so as to specify or adjust an initially automatically determined element location and / or an estimated room shape, and / or manually combine information from multiple estimated room shapes of a room (e.g., separate room shape estimates from different images obtained in the room) to create a final room shape of the room, and / or specify or adjust information of an initially automatically determined final room shape, etc.; providing an input with respect to box 570, which is used as part of a subsequent operation, so as to specify or adjust an initially automatically determined position of a room shape within a generated floor plan and / or specify or adjust the initially automatically determined room shape itself within such a floor plan; providing an input with respect to one or more of boxes 580 and 583 and 585, which is used as part of a subsequent operation, so as to specify or adjust an initially automatically determined information of one or more of the types discussed with respect to those boxes; and / or specifying or adjusting an initially automatically determined pose information (initial pose information or subsequently updated pose information) for one or more of the panoramic images; etc. Additional details of embodiments in which one or more human users provide inputs for further use in additional automated operations of the BOBSDM system are included elsewhere herein.

[0087] Figure 6 FIG. shows an exemplary embodiment of a flowchart of a Building Object Scale Determination Manager (BOBSDM) system routine 600. The routine may be executed, for example, by a Figure 1 BOBSDM system 140, Figure 3 BOBSDM system 340, and / or with reference to Figures 2I to 2L and the BOBSDM systems described elsewhere herein, so as to perform automated operations related to analyzing visual data of images acquired at a building to determine building information including building dimensions. In Figure 6In an exemplary implementation, the images are for a house or other building, but in other implementations, similar analysis may be performed on other types of structures or non-structural locations, and the results may be used in other ways different from those discussed with respect to routine 600, as discussed elsewhere herein.

[0088] The illustrated implementation of the routine begins at block 605, where one or more images of a building (e.g., multiple images taken in multiple rooms of the building) are received. The routine continues to block 610 to analyze each image to identify any visible objects of one or more defined types (e.g., identify the four corners of a doorway; identify at least the bottom and top of a floor cabinet, such as for each column of pixels having visual data of such a cabinet). In block 615, then, for each identified physical object, the routine transforms the points in the image corresponding to the object from the local relative coordinate system of the image to the actual physical coordinate system, including using the assumed physical dimensions (e.g., height) of the defined type of the object to determine the estimated camera height during image acquisition, and using the estimated camera height to determine the actual distance corresponding to the transformed points. In block 620, if more than one physical object is identified in one or more of the images, the routine compares some or all of the determined estimated camera heights (e.g., all heights of all objects of multiple object types across multiple images, multiple heights of different objects of the same type in one or more images, multiple heights of objects of different types in the same image, etc.) and determines whether they meet one or more defined verification criteria. In some implementations and cases, an intermediate aggregated estimated camera height may first be determined for each object type, such as by combining the estimated camera heights of multiple objects of that type in one or more images with multiple such intermediate aggregated estimated camera heights, and then comparing to determine whether they meet one or more defined verification criteria.

[0089] In block 625, the routine then determines whether the verification criteria are met (or, if verification criteria are not used, such as if a single estimated camera height was determined in block 615), and if so, proceeds to execute blocks 630 - 640, otherwise proceeds to block 650. In block 630, the routine uses the compared estimated camera heights to determine a final estimated camera height for one or more images (e.g., by determining their average or other average; by selecting a single estimated camera height, such as the minimum or maximum, etc.), and in block 635, based on the final estimated camera height, determines the building dimensions of the visible building portions in those images (e.g., the length / width and / or height of walls, the dimensions of other visible objects and / or elements, the total room dimensions, the total building dimensions if the image covers the entire building, etc.). In block 640, the routine then optionally uses alternative means to further verify the determined camera height and the resulting building dimensions, such as by using a floor plan generated from the image (optionally waiting, if not yet completed, until its generation), fitting the exterior of the floor plan to the visible exterior of the building in the aerial image, and using the dimension information associated with the aerial image (e.g., GPS data), to determine the resulting overall building dimensions, from which the dimensions of specific rooms can be determined and matched to the corresponding room dimensions determined using the final estimated camera height. If such a comparison is performed and the building and / or room dimensions from the two techniques do not match (e.g., differ by more than a defined threshold amount), then the final building dimension information to be used can be determined in one or more ways, such as selecting the result from one of the two techniques (e.g., a predetermined preference of the two), averaging or otherwise combining the dimensions from the two techniques, requesting a separate inspection and determination (e.g., by an operator user), etc. If it is determined in block 625 that the verification criteria are not met, the routine proceeds to block 650 to perform a technique similar to the one discussed with respect to block 640, thus using instead information based on the estimated camera height.

[0090] After block 640 or block 650, the routine proceeds to block 695 to determine whether to continue, such as until a clear indication to terminate is received, or only continue if a clear indication to continue is received. If it is determined to continue, the routine returns to block 605 to wait and receive additional images to analyze, otherwise proceeds to block 699 and ends.

[0091] Figure 7 An exemplary embodiment of a flowchart for a building information access system routine 700 is shown. The routine can be executed, for example, by Figure 1 a building information access client computing device 175 and its software system (not shown), Figure 3The client computing device 390, and / or a building information access viewer or rendering system as described elsewhere herein, is executed to receive and display the generated floor plan and / or other mapping information (e.g., the determined room structure layout / shape, etc.) for a defined area that optionally includes a visual indication of one or more of the determined image acquisition locations, to obtain and display information about images that match one or more indicated target images, to display additional information (e.g., images) associated with a particular acquisition location in the mapping information, to obtain and display guidance acquisition instructions provided by the BOBSDM system and / or other sources (e.g., regarding other images acquired during the acquisition period and / or regarding the associated building, such as portions of the displayed GUI), to obtain and display an explanation or other description of a match between two or more buildings or properties, etc. In Figure 7 the example, the presented mapping information is for a building (such as the interior of a house), but in other embodiments, other types of mapping information may be presented for other types of buildings or environments and used in other ways, as discussed elsewhere herein.

[0092] The illustrated embodiment of the routine begins at block 705, where instructions or information are received. At block 710, the routine determines whether the instructions or information received at block 705 are to display determined information for one or more target buildings, and if so, proceeds to block 715 to determine whether the instructions or information received at block 705 are to select one or more target buildings using a specified criterion (e.g., at least partially based on the indicated building), and if not, proceeds to block 725 to obtain an indication of the target buildings to be used from the user (e.g., based on a current user selection, such as from a displayed list or other user selection mechanism; based on the information received at block 705, etc.). Otherwise, if it is determined at block 715 to select one or more target buildings from the specified criterion (e.g., at least partially based on the indicated building), the routine proceeds to block 720, where it obtains an indication of one or more search criteria to be used, such as from a current user selection or as indicated in the information or instructions received at block 705, and then searches the stored information about buildings to determine one or more buildings that meet the search criteria or otherwise obtains an indication of one or more such matching buildings, such as at least partially based on building dimension information generated by the BOBSDM system. In the illustrated embodiment, the routine then further selects the best-matching target building from one or more of the returned buildings (e.g., other returned buildings having the highest similarity or other matching rank for the specified criterion, or using another selection technique indicated in the instructions or other information received at block 705), while in other embodiments, the routine may alternatively present multiple candidate buildings that meet the search criteria (e.g., in a sorted order based on the degree of match) and receive the user-selected target building from the multiple candidates.

[0093] After block 720 or block 725, the routine proceeds to block 730 to retrieve a floor plan and / or other generated building mapping information for the target building (e.g., a set of linked images to be used as part of a virtual tour), and optionally associated link information indicating locations surrounding the interior and / or exterior of the building, and / or information about one or more generated explanations or other descriptions of why the target building was selected as matching specified criteria (e.g., based in part or in whole on one or more other indicated buildings), and to select an initial view of the retrieved information (e.g., a view of a floor plan, a particular room shape, a particular image, etc., optionally along with a generated explanation or other description of why the target building was selected as a match if such information is available). In block 740, the routine then displays or otherwise presents the current view of the retrieved information and waits for a user selection in block 745. After the user selection in block 745, if it is determined in block 750 that the user selection corresponds to adjusting the current view of the current target building (e.g., changing one or more aspects of the current view), the routine proceeds to block 755 to update the current view according to the user selection and then returns to block 740 to update the information displayed or otherwise presented accordingly. The user selection and the corresponding update of the current view can include, for example, displaying or otherwise presenting a piece of associated link information selected by the user (e.g., a particular image associated with a displayed visual indication of a determined acquisition location so as to overlay the associated link information on at least some of the previously displayed ones; a particular other image linked to the current image and selectable from the current image using a user-selectable control overlaid on the current image to represent that other image, etc.), and / or changing how the current view is displayed (e.g., zooming in or out; rotating the information if appropriate; selecting a new portion of the floor plan to be displayed or otherwise presented, e.g., some or all of the new portion was previously not visible, or instead the new portion is a subset of the previously visible information, etc.). If it is determined in block 750 that the user selection does not indicate further information about the current target building (e.g., display information about another building, end the current display operation, etc.), the routine proceeds to block 795 and, if the user selection involves such a further operation, returns to block 705 to perform the operation selected by the user.

[0094] If instead it is determined in block 710 that the instruction or other information received in block 705 does not represent information for rendering a building, the routine proceeds to block 760 to determine whether the instruction or other information received in block 705 corresponds to identifying other images (if any) that correspond to one or more indicated target images, and if so, proceeds to blocks 765 - 770 to perform such activities. In particular, in block 765, the routine receives an indication of one or more target images for matching (e.g., from the information received in block 705 or based on one or more current interactions with the user) and one or more matching criteria (e.g., amount of visual overlap), and in block 770, identifies one or more other images (if any) that match the indicated target images, e.g., by interacting with the ICA and / or MIGM systems to obtain the other images. The routine then displays or otherwise provides information about the identified other images in block 770 in order to provide information about them as part of the search results, to display one or more of the identified other images, etc. If it is determined in block 760 that the instruction or other information received in block 705 is not to identify other images that correspond to one or more indicated target images, the routine proceeds to block 775 to determine whether the instruction or other information received in block 705 corresponds to a guidance acquisition instruction for obtaining and providing information about one or more indicated target images (e.g., most recently acquired images) during an image acquisition session, and if so, proceeds to block 780, and otherwise proceeds to block 790. In block 780, the routine obtains information about one or more types of guidance acquisition instructions, e.g., by interacting with the ICA system, and displays or otherwise provides information about the guidance acquisition instructions in block 780, e.g., by overlaying the guidance acquisition instructions on a partial floor plan and / or on the most recently acquired images in a manner discussed in more detail elsewhere herein.

[0095] In block 790, the routine continues to appropriately perform other indicated operations in order to configure parameters to be used in various operations of the system (e.g., at least in part based on information specified by a user of the system, such as a user of a mobile device inside one or more buildings, an operator user of the BOBSDM and / or MIGM systems, etc., including personalizing information display for a particular user according to the particular user's preferences), to obtain and store other information about the system user, to respond to requests for information that has been generated and stored, to perform any housekeeping tasks, etc.

[0096] After box 770 or box 780 or box 790, or if it is determined in box 750 that the user selection does not correspond to the current building, the routine proceeds to box 795 to determine whether to continue, for example until a clear indication to terminate is received, or only proceeds when a clear indication to continue is received. If it is determined to continue (including whether the user makes a selection related to the new building to be presented in box 745), the routine returns to box 705 to wait for additional instructions or information (or if the user makes a selection related to the new building to be presented in box 745, it proceeds directly to box 735, and if not, it advances to step 799 and ends).

[0097] The non-exclusive exemplary embodiments described herein are further described in the following clauses.

[0098] A01. A computer-implemented method for performing automatic operations by one or more computing devices, comprising:

[0099] By the one or more computing devices, obtaining a plurality of images, a predetermined first estimated doorway height, and a predetermined second estimated floor cabinet height, wherein the plurality of images are acquired by a camera device in a plurality of rooms of a house, and during the process of capturing the plurality of images, the camera device is at a constant actual camera height above the floor of the plurality of rooms, and each of the plurality of images is a rectified panoramic image in equirectangular format;

[0100] By the one or more computing devices, analyzing the visual data of the plurality of images to identify a plurality of doorways visible in a plurality of first images of the plurality of images; identifying a plurality of floor cabinets visible in a plurality of second images of the plurality of images, each of the plurality of floor cabinets being installed on the floor of one of the plurality of rooms; and determining the room shapes of the plurality of rooms;

[0101] By the one or more computing devices and based on the analysis, generating an initial floor plan for the house, the initial floor plan including the plurality of house shapes positioned relative to each other and having relative dimensions, wherein the initial floor plan lacks the actual dimensions of the plurality of house shapes;

[0102] By the one or more computing devices, by using information about the positions of the corners of the plurality of doorways in the plurality of first images, and by using the predetermined first estimated doorway height, based on the plurality of doorways, determining a plurality of first estimates of the actual camera height;

[0103] By the one or more computing devices, by combining the determined plurality of first estimates of the actual camera height of the plurality of doorways, generating an aggregated first estimated camera height of the camera device;

[0104] Using the one or more computing devices, determine a plurality of second estimates of the actual camera height based on the plurality of floor cabinets by using information regarding the positions of the tops and bottoms of the plurality of floor cabinets in a plurality of second images and by using the predetermined second estimated floor cabinet height;

[0105] Using the one or more computing devices, generate an aggregated second estimated camera height of the camera device by combining the determined plurality of second estimates of the actual camera height of the plurality of cabinets;

[0106] Using the one or more computing devices, determine the actual dimensions of the plurality of room shapes by: if the aggregated first estimated camera height and the second estimated camera height of the camera device differ by at most a defined threshold amount, generating a final estimate of the actual camera height by combining the aggregated first estimated camera height and the second estimated camera height of the camera device and using the generated final estimate of the actual camera height to determine the wall dimensions of the plurality of rooms, otherwise, fitting the outer walls of the initial floor plan to the exterior of the house visible in the aerial image of the house and using the determined dimension information of the exterior of the house in the aerial image to determine the dimensions of the walls of the plurality of rooms;

[0107] Using the one or more computing devices, update the initial floor plan to include the determined actual dimensions of the plurality of room shapes; and

[0108] Using the one or more computing devices, present an updated floor plan of the house, wherein the determined actual dimensions of the plurality of room shapes are overlaid on the presented updated floor plan.

[0109] A02. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0110] Using the one or more computing devices, obtain a plurality of images, wherein the plurality of images are acquired by a camera device in a plurality of rooms of a building, and during the process of taking the plurality of images, the camera device is at an actual camera height above the floor of the plurality of rooms;

[0111] Using the one or more computing devices, analyze the visual data of the plurality of images to identify a plurality of doorways and a plurality of cabinets visible in the plurality of images, wherein each of the plurality of cabinets is mounted on the floor of one of the plurality of rooms;

[0112] For each of the plurality of doorways and via the one or more computing devices, determine a first estimate of the actual camera height for one or more images in which the doorway is visible by using a predetermined first estimated height of the plurality of doorways, and generate an aggregated first estimated camera height of the camera device by combining the determined first estimates of the actual camera height of the plurality of doorways;

[0113] For each of the plurality of cabinets and via the one or more computing devices, determine a second estimate of the actual camera height for one or more images in which the cabinet is visible by using a predetermined second estimated height of the plurality of cabinets, and generate an aggregated second estimated camera height of the camera device by combining the determined second estimates of the actual camera height of the plurality of cabinets;

[0114] Via the one or more computing devices, verify that the aggregated first estimated camera height and the second estimated camera height of the camera device differ by at most a defined threshold amount;

[0115] Via the one or more computing devices and in response to the verification, determine a final estimate of the actual camera height by combining the aggregated first estimated camera height and the second estimated camera height of the camera device;

[0116] Via the one or more computing devices and based on the determined final estimate of the actual camera height and the visual data of the plurality of images, determine the dimensions of the plurality of rooms by assigning at least a length to at least some of the walls of the plurality of rooms; and

[0117] Via the one or more computing devices, provide the determined dimensions of the plurality of rooms for use with the floor plan of the building Figure 1 for use.

[0118] A03. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0119] Obtain one or more images captured by one or more camera devices in one or more rooms of a building;

[0120] Analyze the visual data of the one or more images to identify a plurality of installed objects visible in the one or more images, the plurality of installed objects including at least one of one or more cabinets installed on one or more floors of the one or more rooms, or one or more doorways;

[0121] For each of the plurality of installed objects, determine an estimate of the actual camera height of at least one of the camera devices during at least one image of the image in which the installed object is visible, by using a predetermined estimated height of the object type of the installed object;

[0122] Verify that the estimates of the actual camera height of the plurality of installed objects thus determined differ by at most a defined threshold amount;

[0123] Generate a final estimated camera height of the one or more camera devices during the one or more images captured by the one or more camera devices, by using the estimates of the actual camera height of the plurality of installed objects thus determined;

[0124] Based on the final estimated camera height and the visual data of the one or more images, and based on the verification, determine the dimensions of the plurality of additional objects visible in the one or more images and different from the plurality of installed objects, by assigning at least one of the determined length or the determined width to each of the plurality of additional objects; and provide the determined dimensions of the plurality of additional objects.

[0125] A04. A computer-implemented method for one or more computing devices to perform an automated operation, comprising:

[0126] Obtain, by the one or more computing devices, a plurality of images captured by one or more camera devices in a plurality of rooms of a building;

[0127] Analyze, by the one or more computing devices, the visual data of the plurality of images to identify a plurality of installed objects of two or more object types visible in at least some of the plurality of images;

[0128] For each of the plurality of installed objects, by the one or more computing devices, determine an estimate of the actual camera height of at least one of the at least one camera device during one or more images in which one or more of the installed objects are visible, by using a predetermined estimated height of the object type of the installed object;

[0129] Verify, by the one or more computing devices, that the estimates of the actual camera height of the plurality of installed objects thus determined differ by at most a defined threshold amount;

[0130] Using the one or more computing devices, during the capture of the one or more images, generate a final estimated camera height of the one or more camera devices by using the estimation of the actual camera height of the determined plurality of installed objects;

[0131] Using the one or more computing devices and based on the final estimated camera height and the visual data of the plurality of images, and based on the verification, determine the dimensions of a plurality of additional objects visible in the plurality of images and different from the plurality of installed objects by assigning at least one of a determined length or a determined width to each of the plurality of additional objects; and

[0132] Using the one or more computing devices, provide the determined dimensions of the plurality of additional objects.

[0133] A05. The computer-implemented method according to any one of clauses A01 - A04, wherein analyzing the visual data of the plurality of images to identify the plurality of doorways and the plurality of floor cabinets includes: using one or more machine learning models trained to identify doorways and floor cabinets.

[0134] A06. The computer-implemented method according to any one of clauses A01 - A05, wherein determining the plurality of first estimates of the actual camera height and determining the plurality of second estimates of the actual camera height includes: using one or more machine learning models trained to use the visual data of doorways and floor cabinets.

[0135] A07. The computer-implemented method according to any one of clauses A01 - A06, wherein the aggregated first estimated camera height and second estimated camera height of the camera device are determined to differ by at most the defined threshold amount, wherein analyzing the visual data of the plurality of images further includes: identifying at least one additional object of at least one additional object type, the at least one additional object type being at least one of an oven, or a dishwasher, or an electric panel, or an air outlet, or a fan blade, or a bed, or a television, or a power outlet, or a wall switchboard, wherein the method further includes: using the visual data of each of the at least one identified additional object of the at least one additional object type to determine one or more third estimates of the actual camera height for each of the additional object types, and wherein generating the final estimate of the actual camera height further includes: combining the determined one or more third estimates of the actual camera height with the aggregated first estimated camera height and second estimated camera height.

[0136] The computer-implemented method as described in any one of clauses A01 - A07 further includes: generating the floor plan of the building based on the visual data of the plurality of images, the floor plan including at least two-dimensional room shapes and relative positions of the plurality of rooms, and wherein providing the determined dimensions of the plurality of rooms includes: presenting overlapping information regarding the dimensions of the building to a visual representation of the floor plan, the overlapping information being based on the determined dimensions of the plurality of rooms.

[0137] A09. The computer-implemented method as described in any one of clauses A01 - A08 further includes:

[0138] fitting, by the one or more computing devices, the outer walls of the floor plan of the building to the exterior of the building visible in the aerial image of the building;

[0139] determining, by the one or more computing devices, the dimensions of the walls of the plurality of rooms using the dimension information of the exterior of the building in the aerial image; and

[0140] determining, by the one or more computing devices, a third estimate of the actual camera height at least in part based on the determined dimensions of the walls of the plurality of rooms and the portions of the walls of the plurality of rooms shown in the visual data of the plurality of images,

[0141] and wherein the verification further includes: determining that the determined third estimate of the actual camera height differs from the aggregated first estimated camera height and second estimated camera height by at most a defined threshold amount.

[0142] A10. The computer-implemented method as described in any one of clauses A01 - A09, wherein each of the plurality of images is a rectified panoramic image in equirectangular format, and wherein analyzing the visual data of the plurality of images to identify the plurality of doorways and the plurality of cabinets includes: using one or more machine learning models trained to identify doorways and cabinets.

[0143] A11. The computer-implemented method as described in any one of clauses A01 - A10, wherein determining at least one of the first estimate of the actual camera height or the second estimate of the actual camera height includes: using one or more machine learning models trained to use visual data of at least one of the doorways or cabinets installed on the floor.

[0144] A12. A computer-implemented method as described in any one of clauses A01 - A11, wherein analyzing the visual data of the plurality of images further comprises: identifying at least one additional object of at least one additional object type, the at least one additional object type being at least one of an oven, or a dishwasher, or an electric panel, or an air outlet, or a fan blade, or a bed, or a television, or a power outlet, or a wall switch plate, wherein the method further comprises: using the visual data of at least one identified additional object of each of the at least one additional object type to determine one or more third estimates of the actual camera height for each of the additional object types, and wherein determining the final estimate of the actual camera height further comprises: combining the determined third estimates of the actual camera height for each of the at least one additional object types.

[0145] A13. A computer-implemented method as described in any one of clauses A01 - A12, wherein the plurality of installed objects includes at least one doorway and at least one cabinet installed on the floor of one of the rooms, and wherein the predetermined estimated height for determining the estimate of the actual camera height includes 80 inches for the doorway and 36 inches for the floor-standing cabinet.

[0146] A14. A computer-implemented method as described in clause A13, wherein the one or more images include a plurality of images taken in a plurality of rooms of the building, wherein the plurality of installed objects further includes a plurality of the doorways and a plurality of the cabinets visible in the plurality of images of the plurality of images,

[0147] wherein the method further comprises, prior to the verification:

[0148] During the taking of at least some of the plurality of images in which the plurality of doorways are visible, generating a first initial aggregated estimated camera height of the one or more camera devices by combining the estimates determined for each of the plurality of doorways; and

[0149] During the taking of at least some of the plurality of images in which the plurality of cabinets are visible, generating a second initial aggregated estimated camera height of the one or more camera devices by combining the estimates determined for each of the plurality of cabinets,

[0150] And wherein verifying that the determined estimates of the actual camera height of the plurality of installed objects differ by at most a defined threshold amount comprises: determining that the estimates of the first initial aggregated estimated camera height and the second initial aggregated estimated camera height differ by at most the defined threshold amount.

[0151] A15. A computer-implemented method as described in any one of clauses A01 - A14, wherein the one or more camera devices are a single camera device maintained at the actual camera height during the capture of the one or more images.

[0152] A16. A computer-implemented method as described in any one of clauses A01 - A15, wherein the building includes a plurality of rooms, wherein the one or more images include a plurality of images captured in the plurality of rooms, wherein determining the dimensions of the plurality of additional objects includes: determining the dimensions of the plurality of rooms, and wherein providing the determined dimensions of the plurality of additional objects includes: presenting overlapping information about the dimensions of the building to a visual representation of the floor plan of the building, the overlapping information being based on the determined dimensions of the building.

[0153] A17. A computer-implemented method as described in clause A16, wherein the two or more object types include floor-standing cabinets installed on one or more floors of one or more of the plurality of rooms and include doorways between the plurality of rooms, wherein a predetermined estimated height for estimating the actual camera height includes 80 inches for the doorways and 36 inches for the floor-standing cabinets, and wherein the plurality of installed objects includes a plurality of doorways and a plurality of floor-standing cabinets visible in a plurality of the plurality of images.

[0154] A18. A computer-implemented method as described in any one of clauses A01 - A17, wherein the plurality of installed objects includes two or more installed objects of each of the two or more object types,

[0155] wherein the method further includes: by the one or more computing devices, before the verification, and for each of the two or more object types, generating an initial aggregated estimated camera height of the one or more camera devices by combining the determined estimates for each of the two or more installed objects for the object type, and

[0156] wherein verifying that the determined estimates of the actual camera height of the plurality of installed objects differ by at most a defined threshold amount includes: comparing the initial aggregated estimated camera heights of the one or more camera devices for the two or more object types.

[0157] A19. A computer-implemented method as described in any one of clauses A01 - A18, wherein the one or more camera devices are a single camera device maintained at the actual camera height during the capture of the plurality of images.

[0158] A20. A computer-implemented method as described in any one of clauses A01 - A19, wherein determining the dimensions of the plurality of additional objects includes determining the dimensions of the plurality of rooms, and wherein providing the determined dimensions of the plurality of additional objects includes: presenting overlapping information regarding the dimensions of the building to a visual representation of the floor plan of the building, the overlapping information being based on the determined dimensions of the plurality of rooms.

[0159] A21. A computer-implemented method as described in any one of clauses A01 - A20, wherein the two or more object types include two or more of an oven, a dishwasher, an electric panel, an air outlet, a fan blade, a bed, a television, a power outlet, or a wall switch plate.

[0160] A22. A computer-implemented method including performing a plurality of steps of an automated operation that implements a technology substantially as disclosed herein.

[0161] B01. A non-transitory computer-readable medium having stored thereon executable software instructions and / or other stored content, the stored executable software instructions and / or other stored content causing one or more computing systems to perform an automated operation that implements a method as described in any one of clauses A01 - A29.

[0162] B02. A non-transitory computer-readable medium having stored thereon executable software instructions and / or other stored content, the stored executable software instructions and / or other stored content causing one or more computing systems to perform an automated operation that implements a technology substantially as disclosed herein.

[0163] C01. One or more computing systems including one or more hardware processors and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform an automated operation that implements a method as described in any one of clauses A01 - A29.

[0164] C02. One or more computing systems including one or more hardware processors and one or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform an automated operation that implements a technology substantially as disclosed herein.

[0165] D01. A computer program adapted to perform a method as described in any one of clauses A01 - A29 when the computer program is run on a computer.

[0166] Aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions. It will be further understood that in some implementations, the functions provided by the above routines can be provided in alternative ways, such as being split among more routines, or combined into fewer routines. Similarly, in some implementations, the illustrated routines can provide more or less functionality than described, such as when other illustrated routines lack or include such functionality respectively, or when the amount of functionality provided changes. Additionally, although various operations may be shown as being performed in a particular manner (e.g., serially or in parallel, or synchronously or asynchronously) and / or in a particular order, in other implementations the operations may be performed in other orders and in other ways. Any data structures discussed above can also be constructed in different ways, such as by splitting a single data structure into multiple data structures and / or by combining multiple data structures into a single data structure. Similarly, in some implementations, the illustrated data structures can store more or less information than described, such as when other illustrated data structures lack or include such information respectively, or when the amount or type of information stored changes.

[0167] From the foregoing, it will be understood that although specific embodiments have been described herein for purposes of illustration, various modifications can be made without departing from the spirit and scope of the invention. Accordingly, the invention is not limited except as by the appended claims and the elements recited therein. Additionally, while certain aspects of the invention may be presented at certain times in certain claim forms, the inventors contemplate the various aspects of the invention in any available claim form. For example, although only some aspects of the invention may be described as being embodied in a computer-readable medium at a particular time, other aspects may also be so embodied.

Claims

1. A computer-implemented method comprising: Obtaining, by one or more computing devices, a plurality of images, wherein the plurality of images are acquired by a camera device in a plurality of rooms of a building, and during the process of capturing the plurality of images, the camera device is located at an actual camera height above the floors of the plurality of rooms; analyzing, by the one or more computing devices, visual data of the plurality of images to identify a plurality of doorways and a plurality of cabinets visible in a plurality of the plurality of images, the plurality of cabinets each mounted on a floor of one of the plurality of rooms; determining, by the one or more computing devices and for each of the plurality of doorways, a first estimate of the actual camera height for one or more of the plurality of images in which the doorway is visible by using predetermined first estimated heights of the plurality of doorways, and generating an aggregated first estimated camera height of the camera device by combining the determined first estimates of the actual camera heights of the plurality of doorways; determining, by the one or more computing devices and for each of the plurality of cabinets, a second estimate of the actual camera height for one or more of the plurality of images in which the cabinet is visible by using a predetermined second estimated height of the plurality of cabinets, and generating an aggregated second estimated camera height of the camera device by combining the determined second estimates of the actual camera height of the plurality of cabinets; verifying, by the one or more computing devices, that an aggregated first estimated camera height and a second estimated camera height of the camera device differ by at most a defined threshold amount; determining, by the one or more computing devices and in response to the verifying, a final estimate of the actual camera height by combining an aggregated first estimated camera height and a second estimated camera height of the camera device; determining, by the one or more computing devices and based on the determined final estimate of the actual camera height and the visual data of the plurality of images, dimensions of the plurality of rooms by assigning at least a length to at least some of the walls of the plurality of rooms; and The determined dimensions of the plurality of rooms are provided, by the one or more computing devices, for use with a floor plan of the building.

2. The computer-implemented method of claim 1 , further comprising: The floor plan of the building is generated based on the visual data of the multiple images, the floor plan including at least two-dimensional room shapes and relative positions of the multiple rooms, and wherein providing the determined sizes of the multiple rooms includes: presenting overlapping information about the size of the building to the visual representation of the floor plan, the overlapping information being based on the determined sizes of the multiple rooms.

3. The computer-implemented method of claim 1 , further comprising: fitting, by the one or more computing devices, exterior walls of the floor plan of the building to an exterior of the building visible in the aerial image of the building; Determining, by the one or more computing devices, dimensions of walls of the plurality of rooms using dimension information of the exterior of the building in the aerial image; as well as determining, by the one or more computing devices, a third estimate of the actual camera height based at least in part on the determined dimensions of the walls of the plurality of rooms and portions of the visual data of the plurality of images showing the walls of the plurality of rooms, And wherein the verifying further comprises determining that the determined third estimate of the actual camera height differs from the aggregated first estimated camera height and second estimated camera height by at most a defined threshold amount.

4. The computer-implemented method of claim 1, wherein: Each of the plurality of images is a straightened panoramic image in an equirectangular format, and wherein analyzing the visual data of the plurality of images to identify the plurality of doorways and the plurality of cabinets comprises using one or more machine learning models trained to identify doorways and cabinets.

5. The computer-implemented method of claim 1, wherein: Determining at least one of the first estimate of the actual camera height or the second estimate of the actual camera height includes: using one or more machine learning models that are trained to use visual data of at least one of a doorway or a cabinet mounted on a floor.

6. The computer-implemented method of claim 1, wherein: Analyzing the visual data of the multiple images also includes: identifying at least one additional object of at least one additional object type, wherein the at least one additional object type is at least one of an oven, or a dishwasher, or an electrical panel, or an air outlet, or a fan blade, or a bed, or a television, or an electrical outlet, or a wall switch plate, wherein the method also includes: using the visual data of at least one identified additional object of each of the at least one additional object type, determining one or more third estimates of the actual camera height for each of the additional object types, and wherein determining the final estimate of the actual camera height also includes: combining the determined third estimates of the actual camera height for each of the at least one additional object type.

7. A system comprising: one or more hardware processors of one or more computing devices; as well as One or more memories having stored instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform automated operations, the automated operations comprising at least: Acquire one or more images captured by one or more camera devices in one or more rooms of a building; analyzing visual data of the one or more images to identify a plurality of installed objects visible in the one or more images, the plurality of installed objects comprising: at least one of the one or more cabinets mounted on one or more floors of the one or more rooms, or one or more doorways; for each of the plurality of mounted objects, determining an estimate of an actual camera height of at least one of the camera devices during capture of at least one of the images by the at least one camera device in which the mounted object is visible, by using a predetermined estimated height of an object type of the mounted object; verifying that the determined estimates of the actual camera heights of the plurality of mounted objects differ by at most a defined threshold amount; said generating a final estimated camera height of one or more camera devices during capture of one or more images by said one or more camera devices by using said determined estimate of said actual camera height of said plurality of mounted objects; determining, based on the final estimated camera height and the visual data of the one or more images, and based on the verifying, sizes of the plurality of additional objects visible in the one or more images and distinct from the plurality of mounted objects by assigning at least one of the determined length or the determined width to each of the plurality of additional objects; and The determined sizes of the plurality of additional objects are provided.

8. The system of claim 7, wherein: The plurality of installed objects include at least one doorway and at least one cabinet mounted on a floor of one of the rooms, and wherein the predetermined estimated heights used to determine the estimate of the actual camera height include 80 inches for the doorway and 36 inches for the floor cabinet.

9. The system of claim 8, wherein: The one or more images include a plurality of images taken in a plurality of rooms of the building, wherein the plurality of installed objects further include a plurality of the doorways and a plurality of the cabinets visible in a plurality of the plurality of images, The stored instructions include software instructions, which, when executed by the at least one hardware processor, cause the at least one computing device to perform further automatic operations, the further automatic operations at least including before the verification: generating a first initial aggregate estimated camera height for the one or more camera devices by combining estimates determined for each of the plurality of doorways during capture of at least some of the plurality of images in which the plurality of doorways are visible; and generating a second initial aggregate estimated camera height for the one or more camera devices by combining the estimates determined for each of the plurality of cabinets during capture of at least some of the plurality of images in which the plurality of cabinets are visible, And wherein verifying that the determined estimates of the actual camera heights of the plurality of installed objects differ by at most a defined threshold amount comprises determining that the estimates of the first initial aggregate estimated camera height and the second initial aggregate estimated camera height differ by at most a defined threshold amount.

10. The system of claim 7, wherein: The one or more camera devices is a single camera device maintained at the actual camera height during capturing of the one or more images.

11. The system of claim 7, wherein: The building comprises a plurality of rooms, wherein the one or more images comprise a plurality of images taken in the plurality of rooms, wherein determining the sizes of the plurality of additional objects comprises determining the sizes of the plurality of rooms, and wherein providing the determined sizes of the plurality of additional objects comprises presenting overlapping information about the sizes of the building to a visual representation of a floor plan of the building, the overlapping information being based on the determined sizes of the building.

12. A non-transitory computer-readable medium having stored content, the stored content causing one or more computing devices to perform an automated operation, the automated operation comprising at least: obtaining, by the one or more computing devices, a plurality of images captured by one or more camera devices in a plurality of rooms of a building; analyzing, by the one or more computing devices, visual data of the plurality of images to identify a plurality of installed objects of two or more object types visible in at least some of the plurality of images; determining, by the one or more computing devices and for each of the plurality of mounted objects, an estimate of an actual camera height of at least one of the camera devices during capture by the at least one camera device of one or more images in which the mounted object is visible in at least some of the images, using a predetermined estimated height for an object type of the mounted object; verifying, by the one or more computing devices, that the determined estimates of actual camera heights of the plurality of mounted objects differ by at most a defined threshold amount; generating, by the one or more computing devices, a final estimated camera height of the one or more camera devices during capture of the one or more images by using the determined estimates of the actual camera heights of the plurality of mounted objects; determining, by the one or more computing devices and based on the final estimated camera height and the visual data of the plurality of images, and based on the verifying, sizes of a plurality of additional objects visible in the plurality of images and distinct from the plurality of mounted objects by assigning at least one of a determined length or a determined width to each of the plurality of additional objects; as well as The determined sizes of the plurality of additional objects are provided, by the one or more computing devices.

13. The non-transitory computer readable medium of claim 12, wherein: The two or more object types include floor-standing cabinets mounted on one or more floors of one or more rooms of the plurality of rooms and include doorways between the plurality of rooms, wherein the predetermined estimated height used to determine the estimate of the actual camera height includes 80 inches for the doorways and 36 inches for the floor-standing cabinets, and wherein the plurality of installed objects includes a plurality of doorways and a plurality of floor-standing cabinets visible in a plurality of the plurality of images.

14. The non-transitory computer readable medium of claim 12, wherein: the plurality of installed objects comprises two or more installed objects of each of the two or more object types, wherein the stored content includes software instructions that, when executed by the one or more computing devices, cause the one or more computing devices to perform further automated operations, the further automated operations comprising at least: generating, by the one or more computing devices, and prior to the verifying, and for each of the two or more object types, an initial aggregate estimated camera height for the one or more camera devices by combining the determined estimates for each of the two or more installed objects of the object type, and Wherein verifying that the determined estimates of the actual camera heights of the plurality of mounted objects differ by at most a defined threshold amount comprises comparing initial aggregated estimated camera heights of the one or more camera devices for the two or more object types.

15. The non-transitory computer readable medium of claim 12, wherein: The one or more camera devices is a single camera device maintained at the actual camera height during capturing of the plurality of images.

16. The non-transitory computer readable medium of claim 12, wherein: Determining the sizes of the multiple additional objects includes determining the sizes of the multiple rooms, and wherein providing the determined sizes of the multiple additional objects includes: presenting overlapping information about the size of the building to a visual representation of the floor plan of the building, the overlapping information being based on the determined sizes of the multiple rooms.

17. The non-transitory computer readable medium of claim 12, wherein: The two or more object types include two or more of an oven, a dishwasher, an electrical panel, an air vent, a fan blade, a bed, a television, an electrical outlet, or a wall switch plate.

18. A computer-implemented method comprising: obtaining, by one or more computing devices, a plurality of images, a predetermined first estimated doorway height, and a predetermined second estimated floor cabinet height, wherein the plurality of images are captured by a camera device in a plurality of rooms of a house, the camera device being located at a constant actual camera height above the floors of the plurality of rooms during the capturing of the plurality of images, wherein each of the plurality of images is a straightened panoramic image in an equirectangular format; analyzing, by the one or more computing devices, visual data of the plurality of images to identify a plurality of doorways visible in a plurality of first images of the plurality of images; identifying a plurality of floor-standing cabinets visible in a plurality of second images of the plurality of images, the plurality of floor-standing cabinets being each mounted on a floor of one of the plurality of rooms; and determining a room shape of the plurality of rooms; generating, by the one or more computing devices and based on the analysis, an initial floor plan for the house, the initial floor plan including the plurality of house shapes positioned relative to one another and having relative sizes, wherein the initial floor plan lacks actual sizes of the plurality of house shapes; determining, by the one or more computing devices, a plurality of first estimates of the actual camera height based on the plurality of doorways by using information regarding the locations of corners of the plurality of doorways in the plurality of first images and by using the predetermined first estimated doorway heights; generating, by the one or more computing devices, an aggregated first estimated camera height for the camera device by combining the determined plurality of first estimates of the actual camera heights for the plurality of doorways; determining, by the one or more computing devices, a plurality of second estimates of the actual camera height based on the plurality of floor cabinets by using information regarding the locations of tops and bottoms of the plurality of floor cabinets in the plurality of second images and by using the predetermined second estimated floor cabinet heights; generating, by the one or more computing devices, an aggregated second estimated camera height of the camera device by combining the determined plurality of second estimates of the actual camera heights of the plurality of cabinets; determining, by the one or more computing devices, actual sizes of the plurality of room shapes by: generating a final estimate of the actual camera height by combining the aggregated first estimated camera height and the second estimated camera height of the camera devices if they differ by at most a defined threshold amount, and determining sizes of walls of the plurality of rooms using the generated final estimate of the actual camera height, otherwise fitting exterior walls of the initial floor plan to exteriors of the house visible in an aerial image of the house, and using the determined size information of the exterior of the house in the aerial image to determine sizes of the walls of the plurality of rooms; updating, by the one or more computing devices, the initial floor plan to include the determined actual dimensions of the plurality of room shapes; and An updated floor plan of the house is presented, by the one or more computing devices, wherein the determined actual sizes of the plurality of room shapes are overlaid on the presented updated floor plan.

19. The computer-implemented method of claim 18, wherein: Analyzing the visual data of the multiple images to identify the multiple doorways and the multiple floor-to-ceiling cabinets includes: using one or more first machine learning models trained to identify doorways and floor-to-ceiling cabinets, and wherein determining the multiple first estimates of the actual camera height and determining the multiple second estimates of the actual camera height includes: using one or more second machine learning models trained to use visual data of doorways and floor-to-ceiling cabinets.

20. The computer-implemented method of claim 18, wherein: The aggregated first estimated camera height and second estimated camera height of the camera device are determined to differ by at most a defined threshold amount, wherein analyzing the visual data of the multiple images also includes: identifying at least one additional object of at least one additional object type, the at least one additional object type being at least one of an oven, or a dishwasher, or an electrical panel, or an air outlet, or a fan blade, or a bed, or a television, or an electrical outlet, or a wall switch plate, wherein the method also includes: using the visual data of at least one identified additional object of each of the at least one additional object type, determining one or more third estimates of the actual camera height for each of the at least one additional object type, and wherein generating a final estimate of the actual camera height also includes: combining the determined one or more third estimates of the actual camera height with the aggregated first estimated camera height and second estimated camera height.

Images