Automated indication for capturing information in a room for building availability assessment

Abstract

Describes techniques for automated operations related to the following operations: analyzing visual data from images captured in a room of a building and optionally additional captured data about the room, evaluating the room layout and other usability information of the rooms of the building and optionally the entire building, and subsequently in one or more further automated ways, using the evaluated usability information, such as to improve navigation in the building. The automated operations may include identifying one or more objects in each of the rooms to be evaluated and one or more target attributes of each object, and generating and providing instructions to cause the capture of images and optionally additional data about those objects and target attributes for further automated analysis, such as instructions provided to automated image capture devices in the room and / or associated users. Once received, the captured images and other data are then analyzed accordingly.

Description

Technical Field

[0001] The following disclosure generally relates to a technology for automatically indicating images and other data captured in a room of a building for evaluating the usability of the room in the building, and subsequently using the evaluated usability information in one or more ways, such as to indicate capturing information about the built-in elements of the room, analyzing the data from the captured images to determine the attributes of the elements, and using this information to evaluate the room layout and other usability information, and using the evaluated room layout and other usability information to improve building navigation and other uses. Background Art

[0002] In various fields and situations, such as building analysis, property inventory, real estate acquisition and development, renovation and retrofit services, general contracting, and others, it may be desirable to view information about the interior of a house, office, or other building without having to physically travel to or enter the building, including determining actual as-built information about the building rather than design information obtained prior to building construction. However, it may be difficult to effectively capture, represent, and use such building interior information, including displaying visual information captured inside the building to a user located at a remote location (e.g., such that the user can fully understand the layout and other details of the interior, including controlling the display in a manner selected by the user). Additionally, although floor plans of a building can provide some information about the layout and other details of the building interior, in some cases, using floor plans in this way has some drawbacks, including that floor plans may be difficult to construct and maintain, difficult to accurately scale and populate with information about the interior of the room, difficult to visualize and use in other ways, etc. Brief Description of the Drawings

[0003] Figures 1A to 1B is a diagram depicting an exemplary building environment and one or more computing systems for use in embodiments of the present disclosure, the computing systems such as for performing automated operations to capture images in a room and subsequently analyzing the visual data of the captured images in one or more ways to generate derived information about the room and the building.

[0004] Figures 2A to 2X shows an example of an automated operation of capturing an image in a room and subsequently analyzing the visual data of the captured image in one or more ways, such as for generating and presenting information about the floor plan of a building and for evaluating the room layout and other usability of the rooms in the building.

[0005] Figure 3 is a block diagram of a computing system showing an embodiment of one or more systems that perform at least some of the techniques described in the present disclosure.

[0006] Figure 4 Shows an example flowchart of an image capture and analysis (ICA) system routine according to an embodiment of the present disclosure.

[0007] Figures 5A to 5C Shows an example flowchart of a mapping information generation manager (MIGM) system routine according to an embodiment of the present disclosure.

[0008] Figures 6A to 6B Shows an example flowchart of a building usability assessment manager (BUAM) system routine according to an embodiment of the present disclosure.

[0009] Figure 7 Shows an example flowchart of a building map viewer system routine according to an embodiment of the present disclosure. Detailed Description

[0010] The present disclosure describes a technique for using a computing device to perform operations related to analyzing visual data from images captured in a room of a building to evaluate the room layout and other usability information of the room and optionally the entire building, and subsequently using the evaluated usability information in one or more further automated ways. The images can include, for example, panoramic images (e.g., in equirectangular or other spherical formats) and / or other types of images (e.g., in rectilinear stereo formats) collected at acquisition locations within or around a multi-room building (e.g., a house, an office, etc.), sometimes referred to herein as "target images". Additionally, in at least some embodiments, the automated operations are performed further without or without using information about the distance from the image acquisition location to walls or other objects in the surrounding building from any depth sensors or other distance measurement devices. In various embodiments, the evaluated room layout and other usability information of one or more rooms of the building can be further used in various ways, such as in combination with generating or annotating corresponding building floor plans and / or other generated information of the building, including controlling a mobile device (e.g., an autonomous vehicle) based on the structural elements of the room, displaying or otherwise presenting in a corresponding GUI (graphical user interface) on one or more client devices via one or more computer networks, etc. The following includes additional details regarding the automated determination and use of room and building usability information, and in at least some embodiments, some or all of the techniques described herein can be performed via automated operations of a building usability assessment manager ("BUAM") system, as further discussed below.

[0011] As described above, the automated operations of the BUAM system can include analyzing visual data of the visual coverage of target images captured in one or more rooms of a building for subsequent use in evaluating the usability of one or more rooms and, in some cases, the entire building. In at least some embodiments, one or more initial target images captured in a room are analyzed to identify various types of information about the room, such as analyzing one or more initial images with room-level visual coverage and identifying specific objects or other elements of the room for which additional data is to be captured, including capturing additional target images that provide more details about the identified objects, including specific target attributes of interest of the objects. In some embodiments, one or more initial target images can provide a wide angle and can include a horizontal coverage of up to 360° around the vertical axis and a coverage between 180° and 360° around the horizontal axis of the room (e.g., one or more panoramic images, such as in a spherical format), and in some embodiments, the additional target images can be more detailed than the initial images (e.g., stereoscopic images, such as in a linear format). After capturing those additional images, the automated operations of the BUAM system can also include parsing additional visual data in the additional visual coverage of the additional images to obtain sufficient data (e.g., above a defined detail threshold, or otherwise meeting one or more defined detail criteria) to allow the assessment of one or more target attributes of interest of each of the identified objects. Once an assessment of the target attributes of the identified objects is available, the automated operations of the BUAM system can also include performing an assessment of each of those identified objects at least in part based on one or more evaluations of one or more target attributes of the objects, such as to estimate how the object contributes to the overall assessment of the room (e.g., the assessment of the usability of the room for an indicated purpose). The automated operations of the BUAM system can also include performing an overall assessment of the room at least in part based on a combination of the assessments of the identified objects in the room, optionally in combination with other information about the room (e.g., the layout of the room, the human traffic flow in the room, etc.). Similarly, in at least some embodiments, the automated operations of the BUAM system can also include performing an overall assessment of the building at least in part based on a combination of the assessments of some or all of the rooms in the building, optionally in combination with other information about the rooms (e.g., the layout of the building, the human traffic flow in the building, etc.). Additionally, in at least some embodiments, for the purposes of the analysis discussed herein, areas outside the building can be considered rooms, such as for defined areas (e.g., terraces, platforms, gardens, etc.) and / or for all surrounding areas (e.g., the outer perimeter of the building), and include identifying and assessing the usability of objects in such "rooms", as well as assessing the target attributes of such objects, and assessing the overall usability of the "room", and including the usability of such "rooms" as part of the assessment of the overall usability of the building.

[0012] As described above, in some embodiments, the automated operation of the BUAM system may include analyzing one or more initial images captured in a room that include room-level visual coverage to identify various types of information about the room. Non-exclusive examples of information that may be automatically determined based on the analysis of the visual data in the one or more initial images include one or more of the following:

[0013] - The presence of specific visible identified objects in the room or other elements of the room for which additional data is to be captured, such as objects that are built-in or otherwise installed in a non-temporary manner and / or other more temporary objects (e.g., objects that can be easily moved, such as furniture or other decorations, etc.);

[0014] - The presence of specific visible target attributes of specific identified objects;

[0015] - The location of some or all of the identified objects in the room (e.g., the location within a specific image, such as using a bounding box and / or a pixel-level mask; relative to the shape of the room, such as relative to the walls and / or the floor and / or the ceiling; relative to a geographical orientation, such as the west wall or the southwest corner, etc.);

[0016] - The type of some or all of the identified objects in the room, such as using object labels or other categories of objects (e.g., windows, doors, chairs, etc.);

[0017] - The type of the room, such as using room labels or other categories of rooms (e.g., bedroom, bathroom, living room, etc.);

[0018] - The layout of the room (e.g., the arrangement of furniture or other items in the room, optionally relative to the shape of the walls and other structural elements of the room);

[0019] - The expected and / or actual traffic flow patterns in the room (e.g., moving between doors and other wall openings in the room and optionally moving to one or more other identified areas of the room, such as relative to the information about the room layout);

[0020] - The intended purpose of the room (e.g., the type of functionality of the room, such as non - exclusive examples based on room type and / or layout, including "kitchen" or "living room", or "cooking" or "personal cooking" or "industrial cooking" in the kitchen, or "collective entertainment" or "personal relaxation" in the living room; and / or the quality of the room and its contents at the time of installation or otherwise when brand - new, such as for room types where quality affects or otherwise is some or all of the intended purposes; and / or the condition of the room and its contents at the current time, such as with respect to the state of repair or disrepair, etc.); and / or the usability of the room, such as based on room layout and / or traffic flow and / or functionality and / or quality and / or condition;

[0021] - Other attributes of the room, such as the degree of "openness" and / or the complexity of the room shape (e.g., cube, L - shaped, etc.) and / or the degree of accessibility, etc.

[0022] In other embodiments, some or all of the above - mentioned information types may not be used for the room usability assessment of some or all rooms and / or the building usability assessment of some or all buildings, and / or may be obtained in other ways (e.g., supplied by one or more users, such as the system operator users participating in data capture in the room of the BUAM system).

[0023] As described above, each room may have one or more elements or other objects that are identified as being of interest (e.g., contributing to the planning assessment of the room), such as based on the type of the room. The identified objects in the room may include objects that are built-in or otherwise installed, non-exclusive examples of which include the following: windows and / or window hardware (e.g., latch mechanisms, opening / closing mechanisms, etc.); doors and / or door hardware (e.g., hinges, door handles, locking mechanisms; peepholes, etc.); installed floors (e.g., tiles, wood, laminate, carpet, etc.); installed countertops and / or backsplashes (e.g., on kitchen islands, kitchen or bathroom counters, etc.); installed wall coverings (e.g., wallpaper, paint, wainscoting, etc.); other built-in structures inside the kitchen island or the front wall of the room (e.g., subsets of recessed or raised floors, grid ceilings, etc.); electrical appliances or gas-powered appliances or other types of powered appliances (e.g., stoves, ovens, microwaves, trash compactors, refrigerators, etc.); light fixtures (e.g., attached to the wall or ceiling); plumbing fixtures (e.g., sinks; bathtubs; showers; toilets; hardware on or inside the sink or bathtub or shower, such as drain pipes, spouts, showerheads, faucets, and other controls, etc.); built-in furniture (e.g., bookshelves, bay window resting areas, etc.); security systems; built-in vacuum systems; air heating and / or cooling systems; water heating systems; various types of pipes or other conduits; various types of electrical wiring; various types of communication wiring; built-in speakers or other built-in electronic devices, etc. Additionally, the objects in the room may include objects that are movable or otherwise temporary, non-exclusive examples of which include the following: furniture; decorations, such as pictures or curtains, etc. In some embodiments, some or all of the identified objects in the room may be automatically determined based on the analysis of visual data of one or more initial images, while in other embodiments, some or all of the identified objects in the room may be automatically determined based on the type of the room (e.g., for a bathroom, a sink and a toilet; for a kitchen, a sink and an oven, etc.), such as based on a predefined list of room types.

[0024] In addition, each recognized object may have one or more target attributes that are recognized as being of interest (e.g., contributing to the planning assessment of the object), such as based on the type of the object and / or the type of the room in which the object is located. Such target attributes may include physical characteristics and / or sub-elements of the object and / or may include various types of functionality or other characteristics of the object. Non-exclusive examples of target attributes include the following: size; material; age; installation type; for door objects, one or more door hardware items, a view of the environment on the other side, or other indications, etc.; for window objects, one or more window hardware items, a view of the environment on the other side, or other indications, etc.; for sinks or bathtubs or showers, the hardware on or inside thereof, the type (e.g., clawfoot bathtub, wall-mounted sink, etc.), functionality (e.g., for a bathtub, including jets), model number, etc.; for stoves, the number of burners, the type of energy used (e.g., electric, gas, etc.), model number, other features or functionality such as a built-in fan, etc.; for other appliances, model number or other type, etc. In some embodiments, some or all of the target attributes of interest for the recognized object may be automatically determined based on an analysis of the visual data of one or more initial images, while in other embodiments, some or all of the target attributes of interest for the object may be automatically determined based on the type of the object and / or the type of the room in which the object is located (e.g., for a sink in a bathroom, the sink hardware and the type of the sink, such as wall-mounted or standalone; for a sink in a kitchen, the size of the sink and the number of bowls and the sink hardware; for a sink in a utility room, the size of the sink and the number of bowls), such as based on a predefined list based on the object type and / or room type.

[0025] In various embodiments, the initial image of the room and additional data about the room can be captured in various ways. In some embodiments, some or all of the initial images of the room can be provided to the BUAM system by another system that has already acquired those images for other purposes, such as by an Image Capture and Analysis (ICA) system and / or a Mapping Information Generation Manager (MIGM) system that uses images of the rooms of a building to generate floor plans and / or other mapping information related to the building, as discussed in more detail below. In other embodiments, some or all of the initial images of the room can be captured by the BUAM system or in response to instructions provided by the BUAM system, such as for an automated image capture device in the room and / or a user in the room (e.g., a BUAM system operator user), where the information indicates the type of initial image to be captured. In a similar manner, in at least some embodiments, some or all of the additional images of the room can be captured by the BUAM system or in response to instructions provided by the BUAM system, such as for an automated image capture device in the room and / or a user in the room (e.g., a BUAM system operator user), where the information indicates the type of additional image to be captured. For example, the BUAM system can provide instructions that identify one or more objects of interest in the room for which additional data is to be captured, and identify one or more target attributes for each of the objects of interest for which additional data is to be captured that meets defined detail criteria or otherwise meets one or more defined detail criteria (or otherwise provides a description of the additional data to be captured for the object that causes sufficient data about one or more target attributes to be captured). In various embodiments, such instructions can be provided in various ways, including being displayed to the user in a GUI of the BUAM system on the user's mobile computing device (e.g., a mobile computing device that acts as an image capture device and is used to capture some or all of the additional images, such as using one or more imaging sensors of the device and optionally additional hardware components of the device, such as lights, one or more IMU (Inertial Measurement Unit) sensors, such as one or more gyroscopes and / or accelerometers and / or magnetometers or other compasses, etc.) or otherwise provided to the user (e.g., superimposed on an image of the room displayed on such a computing device and / or another separate camera device, such as providing dynamic augmented reality instructions to the user as the image changes in response to movement of the device, and / or providing static instructions to the user on a previously captured image, and optionally having visual markers on one or more images of visible objects and / or target attributes), or alternatively provided to an automated device that acquires additional images in response to the instructions.

[0026] Additionally, the automated operation of the BUAM system may further include analyzing the visual data of the images to verify that they include sufficient details to meet a defined detail threshold or otherwise meet one or more defined detail criteria, and initiating further automated operations in response to the verification activity. For example, the visual data of one or more initial images may be analyzed to determine, for each of some or all of the objects of interest, whether the one or more initial images already include sufficient data regarding one or more target attributes of interest of the object. If so, the one or more initial images may be used to perform the assessment of the object and its one or more target attributes instead of the visual data of additional images that would otherwise have been captured and used. However, if not, then the BUAM system may initiate capturing one or more additional images to provide sufficient data regarding one or more target attributes of interest of the object. Additionally, once one or more additional images are captured for an object (e.g., additional images for each of one or more target attributes of the object), the additional visual data of the one or more additional images may be similarly analyzed to verify that they include visual data of one or more target attributes with sufficient details to meet a defined detail threshold or otherwise meet one or more defined detail criteria, and in some embodiments, to verify that the one or more additional images actually belong to the correct object and / or correct target attributes (e.g., comparing the visual terms of the additional images with the corresponding visual data of one or more initial images of the object). If so, those additional images may be verified as being usable for the assessment of those target attributes and the associated assessment of the object. However, if not, then the BUAM system may initiate recapturing one or more new additional images to replace one or more previous additional images having verification issues and provide sufficient details of one or more target attributes of interest of the object (or initiate other corrective actions, such as requesting additional details regarding one or more target attributes in text or other form). In various embodiments, the defined detail threshold or other one or more defined detail criteria may have various forms, such as other metrics indicating the minimum pixel amount or resolution in an image of a target attribute or object that is the subject of the image, the minimum illumination level, the maximum amount of blur, or other metrics of the clarity of the visual data, etc.

[0027] Additionally, in at least some embodiments, whether supplementing or replacing the corresponding additional images, and in response to corresponding instructions provided to the user and / or automated devices by the BUAM system, the automated operations of the BUAM system to capture additional data may also include capturing other types of data in addition to the additional images. For example, the BUAM system may provide instructions to an image acquisition device (e.g., a mobile computing device having one or more imaging sensors and other hardware components) to capture data regarding specific objects and / or target attributes using other sensors (e.g., IMU sensors, microphones, GPS or other location sensors, etc.), and provide the other captured additional data to the BUAM system for further analysis, and / or may provide instructions to the user to obtain and provide additional data in a form other than visual data (e.g., text answers to questions and / or recorded voice answers), such as for aspects of objects and / or target attributes that may not be easily determined from visual data (e.g., for an object or target attribute, material, age, size, exact location, installation technique / type, model, one or more types of functionality, etc.). This type of captured additional data, in addition to the captured additional images, can be used in various ways, including as part of the assessment of target attributes and the associated evaluation of the object, as discussed in more detail elsewhere in this document. In various embodiments, the details of interest to be obtained for the target attribute and / or object, as well as the associated instructions provided by the BUAM system, may be automatically determined by the BUAM system in various ways, such as based on a predefined list or other description of the type of target attribute and / or object type and / or the type of room in which the object and its one or more target attributes are located.

[0028] After the BUAM system has obtained the captured additional data of the objects of interest and the target attributes in the room (regardless of the visual data from the captured additional images, other captured additional data, or the visual data from one or more initial room-level images), the additional data can be analyzed in various ways to assess each of the target attributes and evaluate each of the objects, such as at least partially based on the assessment of one or more target attributes of the object. In at least some embodiments, each of the target attributes can be assessed by the BUAM system relative to one or more defined assessment criteria, such as in a manner specific to the type of the target attribute. For example, a target attribute can be assessed relative to one or more factors, non-exclusive examples of such factors including the following: material; age; size; exact location; installation technique / type; model; one or more types of functionality; the quality of the target attribute at the time of installation or when new; the condition of the target attribute at the current time, such as the state relative to repair or disrepair, etc., and factors specific to a particular type of object and / or target attribute (e.g., for a door and / or door lock and / or window latch, the degree of strength and / or other anti-intrusion protection; for a door and / or door handle and / or window, the degree of decorative appeal, etc.). If multiple factors are assessed individually, then in at least some embodiments, an overall assessment of the target attribute can be further determined, such as via weighted average or other combination techniques, and optionally where the weights vary based on the specific factors. It should be understood that in some embodiments, the assessment of a specific target attribute of an object relative to a specific factor can be provided by one or more users and used in combination with other automatically determined assessments of other target attributes of the object relative to the evaluation of the object.

[0029] Additionally, after one or more target attributes of an object have been evaluated, the evaluation of the object can be automatically determined by the BUAM system relative to one or more object evaluation criteria, whether the same as or different from the evaluation criteria for the one or more target attributes of the object. In at least some embodiments, the evaluation of an object in a room is at least partially performed relative to the usability of the room for its intended purpose, such as to estimate the contribution of the object to meeting the intended purpose of the room. As a non-exclusive example, a sink with high quality, condition, and functionality in the master bathroom (e.g., at least partially based on the sink fixture or other sink hardware) can greatly contribute to the evaluation of the master bathroom and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the master bathroom, such as in cases where a luxurious environment is part of the intended purpose), but may contribute little or nothing (or even a negative contribution) to the evaluation of a utility room and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the utility room, such as based on practical functionality as part of the intended purpose), if there is a bathroom present. More generally, each of the objects can be evaluated by the BUAM system at least partially based on combining the evaluations of one or more of the one or more target attributes of the object and optionally relative to one or more additional defined object evaluation criteria, such as in a manner specific to the type of the object. For example, an object can be evaluated relative to one or more factors, non-exclusive examples of such factors including the following: material; age; size; exact location; installation technique / type; model; one or more types of functionality; quality of the object at the time of installation or when new; condition of the object at the current time, such as relative to the state of repair or disrepair, etc., and factors specific to a particular type of object and / or target attribute (e.g., for a door, the degree of strength and / or other anti-intrusion protection, the degree of decorative appeal, etc.). If multiple factors are evaluated separately, in at least some embodiments, an overall evaluation of the object can be further determined, such as via weighted average or other combination techniques, and optionally where the weights vary based on the specific factors. It should be understood that in some embodiments, the evaluation of a specific object in a room relative to a specific factor, or more generally relative to the usability for the intended purpose of the room, can be provided by one or more users and used in combination with other automatically determined evaluations of other objects in the room as part of the evaluation of the room. As described above, the intended purpose of the room can be at least partially based on the type of the room, and the usability of the room for the intended purpose can be based on one or more factors, such as functionality, quality, condition, etc.

[0030] Additionally, after evaluating one or more objects of interest in a room, the BUAM system can automatically determine an evaluation of the room relative to one or more room evaluation criteria (whether the same as or different from one or more object evaluation criteria of the objects in the room). In at least some embodiments, the evaluation of the room is performed at least in part relative to the usability for the intended purpose of the room, such as at least in part based on the evaluation of one or more objects of interest in the room and their estimated contribution to meeting the intended purpose of the room. As a non-exclusive example, a sink with high quality, condition, and functionality in a master bathroom (e.g., at least in part based on the sink fixture or other sink hardware) can greatly contribute to the evaluation of the master bathroom and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the master bathroom, such as based on a luxurious environment as part of that intended purpose), but may contribute little or not at all (or even negatively) to the evaluation of a utility room and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the utility room, such as based on practical functionality as part of that intended purpose) if there is a bathroom present. As another non-exclusive example, the evaluation of a room can be at least in part based on the compatibility of the fixtures and / or other objects within the room, such as sharing the same style, quality, etc. More generally, each of one or more rooms in a building can be evaluated by the BUAM system at least in part by combining one or more evaluations of one or more objects of interest in the room and optionally relative to one or more additional defined room evaluation criteria, such as in a manner specific to the type of the room. For example, a room can be rated relative to one or more factors, non-exclusive examples of such factors including the following: size; layout; shape; traffic flow; materials; age; quality of the room at the time of installation or when brand new; condition of the room at the current time, such as relative to the state of repair or disrepair, etc., and factors specific to a particular type of room (e.g., for a master bathroom or kitchen, degree of luxury and / or quality; for a utility room or corridor, degree of functionality or usability, etc.). If multiple factors are rated separately, in at least some embodiments, an overall evaluation of the room can be further determined, such as via weighted average or other combination techniques, and optionally where the weights vary based on the specific factors. It should be understood that in some embodiments, the evaluation of a particular room relative to a particular factor, or more generally relative to the intended purpose of the room, can be provided by one or more users and used in combination with other automatically determined evaluations of other rooms in the same building as part of the overall evaluation of the building.

[0031] Additionally, after evaluating the rooms of a multi-room building, the overall evaluation of the building can be automatically determined by the BUAM system relative to one or more building evaluation criteria (whether the same as or different from the room evaluation criteria of the rooms of the building). In at least some embodiments, the evaluation of the building is performed at least in part relative to the usability for the intended purpose of the building, such as at least in part based on the evaluation of the rooms in the building and the satisfaction of the separately evaluated intended purposes of the rooms, optionally in combination with additional factors such as the overall layout of the building and / or the expected traffic flow through the building. As a non-exclusive example, if the building is a single-family house, a sink with high quality, condition, and functionality in the bathroom (e.g., at least in part based on the sink fixture or other sink hardware) can contribute significantly to the overall evaluation of the building and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the house), but if the building is a warehouse, it may contribute little or nothing (or even a negative contribution) to the overall evaluation of the building and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the warehouse, such as based on the usability as part of that intended purpose). More generally, the building can be evaluated by the BUAM system at least in part based on combining one or more evaluations of some or all of the rooms in the room and optionally relative to one or more additional defined building evaluation criteria, such as in a manner specific to the type of the building. For example, a building can be rated relative to one or more factors, non-exclusive examples of such factors including the following: size; layout (e.g., based on the floor plan of the building); shape; traffic flow; materials; age; the quality of the building at the time of installation or when brand new; the condition of the building at the current time, such as relative to the state of repair or disrepair, etc., and factors specific to a particular type of building (e.g., for a house or an office building, the degree of luxury and / or quality; for a warehouse or storage facility, the degree of functionality or usability, etc.). If multiple factors are rated, in at least some embodiments, the overall evaluation of the building can be further determined, such as via weighted average or other combination techniques, and optionally where the weights vary based on specific factors. It should be understood that in some embodiments, the evaluation of a particular building relative to a specific factor, or more generally relative to the usability for the intended purpose of the building, can be provided by one or more users and used in combination with other automatically determined evaluations of other relevant buildings in a group as part of the overall evaluation of the group of buildings.

[0032] As described above, some or all of the information, relative to the information from the additional data captured for assessing a target attribute and / or evaluating an object and / or evaluating a room, may be based on the analysis of visual data in one or more initial room-level images and / or one or more additional images. As part of the automated operation of the BUAM system, in at least some embodiments, the described techniques may include using one or more trained neural networks or other techniques to analyze the visual data of one or more initial images and / or additional images. As non-exclusive examples, such techniques may include one or more of the following: using a trained neural network or other analysis technique (e.g., a convolutional neural network) to take as input some or all of the images of a room and identify objects of interest in the room, such objects may include, for example, wall structural elements (e.g., windows and / or skylights; passages into and / or out of the room, such as doorways and other openings in the wall, stairs, corridors, etc.; the boundaries between adjacent walls; the boundary between a wall and a floor; the boundary between a wall and a ceiling; corners (or geometric vertices) where at least three surfaces or planes intersect, etc.), other fixed structural elements (e.g., countertops, bathtubs, sinks, islands, fireplaces, etc.); using a trained neural network or other analysis technique to take as input some or all of the one or more images of a room and determine the room shape of the room, such as a 3D point cloud (where multiple 3D data points correspond to positions on the walls and optionally the floor and / or ceiling) and / or fully or partially connected planar surfaces (corresponding to some or all of the walls and optionally the floor and / or ceiling) and / or wireframe structure lines (e.g., to show one or more of the boundaries between walls, the boundary between a wall and a ceiling, the boundary between a wall and a floor, the outline of a doorway and / or other inter-room wall openings, the outline of a window, etc.); using a trained neural network or other analysis technique (e.g., a deep learning detector model or other type of classifier) to take as input some or all of the one or more images of a room (and optionally the determined room shape of the room) and determine the positions of the detected objects and other elements in the room (e.g., relative to the shape of the room, based on performing object detection to generate bounding boxes around elements or other objects in one or more images, based on performing object segmentation to generate pixel-level masks identifying the pixels representing elements or other objects in one or more images, etc.); using a trained neural network or other analysis technique (e.g., a convolutional neural network) to take as input some or all of the one or more images of a room and determine the object labels and / or object types of those elements or other objects (e.g., window, doorway, etc.); using a trained neural network or other analysis technique to take as input some or all of the one or more images of a room and determine the room type and / or room label of the enclosed room (e.g., living room, bedroom, bathroom, kitchen, etc.);Use a trained neural network or other analysis techniques to obtain some or all of one or more images of a room (e.g., a panoramic image with 360° horizontal visual coverage) as input and determine the layout of the room; use a trained neural network or other analysis techniques to obtain some or all of one or more images of a room as input and determine the expected traffic flow in the room; use a trained neural network or other analysis techniques to obtain some or all of one or more images of a room as input (and optionally information about the room type / label and / or layout and / or traffic flow) and determine the expected purpose of the enclosed room; use a trained neural network or other analysis techniques to obtain some or all of one or more images of a room as input and identify the visible target attributes of the objects of interest; use a trained neural network or other analysis techniques to obtain some or all of one or more images of a room as input and determine whether one or more visible target attributes have sufficient detail in the visual data to meet a defined detail threshold or otherwise meet one or more defined detail criteria; use a trained neural network or other analysis techniques to obtain some or all of one or more images of a room as input and determine whether one or more visible objects have sufficient detail in the visual data to meet a defined detail threshold or otherwise meet the defined detail criteria; use a trained neural network or other analysis techniques to obtain some or all of one or more images of an object (and optionally additional captured data about one or more target attributes and / or the object) as input and assess each of the one or more target attributes at least in part based on the visual data of the one or more images; use a trained neural network or other analysis techniques to obtain one or more images of an object (and optionally additional captured data about the object and / or its room, including identifying the expected purpose of the room) as input and evaluate the object at least in part based on the visual data of the one or more images; use a trained neural network or other analysis techniques (e.g., using rule-based decision-making, such as where predefined rules are specified by one or more BUAM system operator users or otherwise determined) to obtain an assessment of one or more target attributes of an object (and optionally additional captured data about the object and / or its room, including identifying the expected purpose of the room) as input and evaluate the object at least in part based on the assessment of the target attributes; use a trained neural network or other analysis techniques to obtain an assessment of one or more objects in a room (and optionally additional captured data about the objects and / or the room, including identifying the expected purpose of the room) as input and evaluate the room at least in part based on the evaluation of the objects;Use a trained neural network or other analysis techniques to obtain an assessment of one or more rooms in a building (and optionally additional captured data about the room and / or its building, including identifying the intended purpose of the building) as input and evaluate the building at least in part based on the assessment of the room, etc. In different embodiments, such neural networks may use, for example, different detection and / or segmentation frameworks, and in different embodiments, may otherwise belong to various types and may be trained by the BUAM system on a dataset corresponding to the determined type of the neural network execution before use. In some embodiments, the acquisition metadata of such images may further be used to determine one or more of the types of information discussed above, such as using data from an IMU (Inertial Measurement Unit) sensor on the acquisition camera or other associated device as part of performing SLAM (Simultaneous Localization and Mapping) and / or SfM (Structure from Motion) and / or MVS (Multi-View Stereo) analysis, or otherwise determining the acquisition pose information of the images in the room, as discussed elsewhere in this document.;

[0033] Additional details regarding automated operations are included below, which in at least some embodiments may be performed by the BUAM system to collect and analyze visual data of the visual coverage from target images captured in one or more rooms of a building, and / or use the information from the analysis to evaluate the usability of the room. For example, examples regarding Figures 2P to 2X and their associated descriptions and some corresponding additional details are included in Figures 6A to 6B and elsewhere in this document.

[0034] As described above, after evaluating the usability of one or more rooms of a building, and optionally further evaluating the overall usability of the building, based at least in part on an analysis of visual data from images captured in one or more rooms, the automated operation of the BUAM system may further include using the evaluated room and / or building usability information in one or more further automated ways. For example, as discussed in more detail elsewhere herein, such evaluation information may be associated with a floor plan and / or other generated mapping information of one or more rooms and / or the building, and used to improve the automated navigation of a mobile device (e.g., a semi-autonomous or fully autonomous vehicle) through the building, at least in part based on the determined evaluation of the rooms and the building (e.g., based on room layout, traffic flow, etc.). In some embodiments, such information regarding room and / or building and / or object evaluation and the determination of target attributes of objects may further be used in additional ways, such as displaying the information to users to assist them in navigating the room and / or building, or for other uses by the users. In some embodiments, such information regarding one or more room and / or building and / or object evaluation and the determination of target attributes of objects may also be used in other ways, such as automatically identifying areas of improvement or renovation in the building (e.g., in a particular room, and / or with respect to a particular object and / or its target attributes), automatically evaluating the price and / or value of the building (e.g., based on a comparison with other buildings having a similar evaluation of overall building usability with respect to the overall intended purpose of the building and / or a similar evaluation of room usability with respect to the intended purpose of some or all of the rooms of the building), etc. It should be understood that in other embodiments, various other uses of the evaluation information may be obtained.

[0035] In various embodiments, the described techniques provide various benefits, including allowing floor plans of multi-room buildings and other structures to be automatically enhanced with information regarding the assessment of rooms in a building and / or regarding the overall assessment of the building, and optionally the assessment of specific objects in the rooms, the determination of target attributes of the building. In some embodiments, such information regarding room and / or building and / or object assessment and the determination of target attributes of the objects can also be used in additional ways, such as automatically identifying areas of improvement or renovation in the building (e.g., in a specific room, and / or with respect to a specific object and / or its target attributes), automatically assessing the price and / or value of the building, automatically ensuring that the desired type of information is captured and used (e.g., at least in part by associated users who are not experts or otherwise trained in such information capture), and the like. Additionally, such automated techniques allow for the determination of such building, room, and object information more quickly than prior existing techniques and, in at least some embodiments, with higher accuracy, including by using information collected from an actual building environment (as opposed to plans regarding how a building should theoretically be constructed), and enabling the capture of changes to structural elements or other portions of the building that occur after the building is initially constructed. The described techniques further provide the benefit of allowing for improvements in the automatic navigation of a building by a mobile device (e.g., a semi-autonomous or fully autonomous vehicle) at least in part based on the determined assessment of the rooms and the building (e.g., based on room layout, traffic flow, etc.), including a significant reduction in the computational power and time required to otherwise learn the building layout. Additionally, in some embodiments, the described techniques can be used to provide an improved GUI where a user can more accurately and quickly obtain information regarding the interior of the building (e.g., for use in navigating within the interior), including in response to a search request, as part of providing personalized information to the user, as part of providing a value estimate and / or other information regarding the building to the user, and the like. The described techniques also provide various other benefits, some of which are further described elsewhere herein.

[0036] As described above, in at least some embodiments and scenarios, some or all of the images collected for a building may be panoramic images collected at one of a plurality of collection locations within or around the building, such as generated from one or more of the following to generate a panoramic image at each such collection location: video captured at the collection location (e.g., 360° video captured by rotating a smart phone or other mobile device held by a user at the collection location), or multiple images captured from the collection location in multiple directions (e.g., by rotating a smart phone or other mobile device held by a user at the collection location; from automated rotation of the device at the collection location, such as on a tripod at the collection location, etc.), or capturing all of the image information simultaneously for a specific collection location (e.g., using one or more fisheye lenses), etc. It should be understood that in some scenarios, such panoramic images may be represented in a spherical coordinate system and provide up to 360° coverage around the horizontal and / or vertical axes (e.g., 360° coverage along the horizontal plane and around the vertical axis), while in other embodiments, the collected panoramic images or other images may include less than 360° horizontal and / or vertical coverage (e.g., for images with a width-to-height ratio greater than typical, such as at or exceeding 21:9 or 16:9 or 3:2 or 7:5 or 4:3 or 5:4 or 1:1, including so-called "ultra-wide" lenses and the resulting ultra-wide images). Additionally, it should be understood that a user who is permitted to view such a panoramic image (or other images with sufficient horizontal and / or vertical coverage such that only a portion of the image is displayed at any given time) may move the viewing direction to different orientations within the panoramic image to indicate rendering different subsets of the images (or "views") within the panoramic image, and in some scenarios, such panoramic images may be represented in a spherical coordinate system (including converting the image to a planar coordinate system for rendering a specific view if the panoramic image is represented in a spherical coordinate system and rendered, such as for a stereoscopic image view before display). Additionally, collection metadata regarding the capture of such panoramic images may be obtained and used in various ways, such as data collected from an IMU (inertial measurement unit) sensor or other sensors of the mobile device when the mobile device is carried by the user or otherwise moved between collection locations. Non-exclusive examples of such collection metadata may include one or more of the following: collection time; collection location, such as GPS coordinates or other location indicators; collection direction and / or orientation; relative or absolute collection order of multiple images collected for or otherwise associated with the building, etc., and in at least some embodiments and scenarios, such collection metadata may also optionally be used as part of determining the collection location of the image, as discussed further below. The following includes additional details regarding automated operations involved in collecting images and optionally collection metadata by one or more devices implementing an Image Capture and Analysis (ICA) system, including regarding Figures 1A to 1B and Figures 2A to 2D andFigure 4 and additional details elsewhere in this document.

[0037] Similarly as described above, in various embodiments, the shape of a room in a building can be automatically determined in various ways, including determination at a time prior to the automated determination of the acquisition locations of specific images within the building. For example, in at least some embodiments, a Mapping Information Generation Manager (MIGM) system can analyze various images acquired within and around a building in order to automatically determine the room shape (e.g., 3D room shape, 2D room shape, etc.) of the rooms in the building and automatically generate a floor plan of the building. As an example, if multiple images are acquired within a specific room, those images can be analyzed to determine the 3D shape of the room in the building (e.g., to reflect the geometry of the surrounding structural elements of the building), and the analysis can include, for example, automated operations to "register" the camera positions of the images in a common reference frame in order to "align" the images, and estimate the 3D positions and shapes of objects in the room, such as determining features visible in the content of such images (e.g., to determine the direction and / or orientation of the acquisition device when taking a specific image, the path the acquisition device traveled through the room, etc., such as using SLAM techniques for multiple video frame images and / or other SfM techniques for a set of "dense" images separated by at most a defined distance (such as 6 feet) to generate a 3D point cloud that includes 3D points along at least some of the walls of the room and the ceiling and floor of the room and optionally has 3D points corresponding to other objects in the room, etc.) and / or determining and aggregating information about the planes of the detected features and the normal (orthogonal) directions of which planes to identify planar surfaces that may be the locations of walls and other surfaces of the room and connecting various possible wall locations (e.g., using one or more constraints, such as having a 90° angle between walls and / or between a wall and the floor, as part of the so-called "Manhattan world assumption") and forming an estimated room shape of the room. After determining the estimated room shapes of the rooms in the building, in at least some embodiments, the automated operations can further include positioning multiple room shapes together to form a floor plan of the building and / or other relevant mapping information, such as by connecting the various room shapes. Thus, such a building floor plan can have associated room shape information, and in various embodiments, can have various forms, such as a 2D (two-dimensional) floor map of the building (e.g., an orthographic top view or other top view of a schematic floor map that does not include or display height information) and / or a 3D (three-dimensional) or 2.5D (two-and-a-half-dimensional) floor map model of the building that displays height information.As part of the automated analysis of visual data of one or more target images, the automated operations can include determining the acquisition location and optionally the orientation of the target image captured in a room of a house or other building (or in another defined area), and using the determined acquisition location and optionally the orientation of the target image to further analyze the visual data of the target image. The combination of the acquisition location and orientation of the target image is sometimes referred to herein as the "acquisition pose" or "acquisition location" of the target image or simply as "pose" or "location". Additional details are included below regarding the automated operations involved in determining room shapes and combining room shapes to generate a floor plan by one or more devices implementing the MIGM system, including regarding. Figures 1A to 1B and Figures 2E to 2M as well as Figures 5A to 5C and additional details elsewhere in this document.

[0038] For illustrative purposes, some embodiments are described below in which specific types of information are collected, used, and / or presented in a specific manner and by using specific types of devices for a specific type of structure - however, it will be understood that the described techniques may be used in other ways in other embodiments, and thus the present invention is not limited to the provided exemplary details. As a non-exclusive example, although in some examples a specific type of object and room in a house are discussed for evaluation, it should be understood that in other embodiments, other types of evaluations may be similarly generated, including for a building separate from the house (or other structure or layout). As another non-exclusive example, although in some examples, specific types of instructions are provided in a specific manner to obtain specific types of data, in other embodiments, other types of instructions may be used and other types of data may be collected in other ways. Additionally, the term "building" as used herein refers to any structure that is partially or fully enclosed and typically but not necessarily encompasses one or more rooms that visually or otherwise divide the interior space of the structure. Non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, condominiums, office buildings, commercial buildings or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.), and the like. As used herein with reference to the interior of a building, a collection location, or other location (unless the context clearly indicates otherwise), the term "collect" or "capture" may refer to any recording, storage, or entry of media, sensor data, and / or other information related to the spatial characteristics and / or visual characteristics and / or other perceivable characteristics or subsets thereof within the interior of the building, such as by a recording device or by another device that receives information from the recording device. As used herein, the term "panoramic image" may refer to a visual representation that is based on, includes, or can be divided into multiple discrete component images that originate from substantially similar physical locations in different directions and depicts a larger field of view than any one of the discrete component images alone depicts, including an image from a physical location having a wide enough viewing angle to include an angle beyond what can be perceived by a person gazing in a single direction (e.g., greater than 120° or 150° or 180°, etc.). As used herein, the term "series" of collection locations generally refers to two or more collection locations, each of which is accessed at least once in a corresponding order, regardless of whether other non-collection locations are accessed between them, and regardless of whether the access to the collection locations occurs during a single continuous time period or at multiple different times, or is performed by a single user and / or device or by multiple different users and / or devices. Additionally, for illustrative purposes, various details are provided in the drawings and the text, but these details are not intended to limit the scope of the present invention. For example, the dimensions and relative positions of elements in the drawings are not necessarily drawn to scale, and some details are omitted and / or provided more prominently (e.g., via sizing and positioning) to enhance readability and / or clarity. Additionally, the same reference numerals may be used in the drawings to identify similar elements or actions.

[0039] Figure 1A are example block diagrams of various computing devices and systems that may participate in the described technology in some embodiments. In particular, in Figure 1A is shown a panoramic image 165 generated by an internal capture and analysis (“ICA”) system 160 executing on one or more server computing systems 180 in this example, such as with respect to one or more buildings or other structures, and wherein optionally inter-image orientation links have been generated for at least some image pairs, Figure 1B shows an example of such panoramic image acquisition locations 210 for a particular house 198 (e.g., inter-image relative orientation links 215-AB, 215-AC, and 215-BC between image pairs from acquisition locations 210A and 210B, 210A and 210C, and 210B and 210C, respectively), as discussed further below, and additional details related to the automated operation of the ICA system are included elsewhere in this document, including details regarding Figure 4 In at least some embodiments, at least some of the ICA system may be executed in part on a mobile computing device 185 (either supplementing or replacing the ICA system 160 on one or more server computing systems 180), such as in an optional ICA application 154, to control the acquisition of target images and optionally additional non-visual data by the mobile computing device and / or one or more optional separate camera devices 186 operating in cooperation with the mobile computing device (e.g., in the same room), as discussed further regarding Figure 1B The MIGM (Mapping Information Generation Manager) system 160 is further executed on Figure 1A one or more of the server computing systems 180 in Figures 2M to 2O (referred to herein as “2-O” for clarity) shows an example of such a floor plan, as discussed further below, and additional details related to the automated operation of the MIGM system are included elsewhere in this document, including details regarding Figures 5A to 5B of.

[0040] Figure 1AFurther shown is a BUAM (Building Usability Assessment Manager) system 140 that executes on one or more server computing systems 180 to automatically analyze visual data from images 144 captured in rooms of a building (e.g., at least in part based on panoramic image 165) to evaluate the usability of rooms in the building and then use the evaluated usability information in one or more ways, including generating and using various usability information 145 during operation of the BUAM system (e.g., information about objects in the room and their target attributes, images of the objects and their target attributes, room layout data, ratings of target attributes, assessments of objects and / or rooms and / or buildings, etc.). In at least some embodiments and scenarios, one or more users of the BUAM client computing device 105 may further interact with the BUAM system 140 via one or more networks 170, such as to assist with some of the automated operations of the BUAM system. Additional details related to the automated operations of the BUAM system are included elsewhere herein, including details about Figures 2P to 2X and Figures 6A to 6B In some embodiments, the ICA system and / or the MIGM system and / or the BUAM system 140 may execute on the same one or more server computing systems, such as if multiple or all of those systems are operated by a single entity or otherwise coordinated to execute (e.g., some or all of the functions of those systems are integrated together into a larger system), while in other embodiments, the BUAM system may operate separately from the ICA and / or MIGM systems (e.g., without using any information generated by the ICA and / or MIGM systems).

[0041] One or more users (not shown) of one or more client computing devices 175 may further interact with the BUAM system 140 and optionally the ICA system and / or the MIGM system via one or more computer networks 170, such as to assist with the automated operations of one or more systems and / or obtain information generated by one or more of the systems (e.g., the captured images; the generated floor plans, such as having information about the generated object and / or room and / or building assessments superimposed on or otherwise associated with the floor plan, and / or having information about one or more of the captured images superimposed on or otherwise associated with the floor plan); information about the generated object and / or room and / or building assessments, etc.) and optionally interact with it, including optionally changing between a floor plan view and a view of a specific image at the acquisition location within or near the floor plan; changing the horizontal and / or vertical viewing directions according to which the corresponding view of the panoramic image is displayed, such as determining the part of the panoramic image that the current user viewing direction points to, etc. Additionally, although Figure 1Ais not described, but a floor plan (or a portion thereof) may be linked to or otherwise associated with one or more other types of information, including multiple associated sub-floor plans of different floors or levels of a multi-story building or other multi-level structure that are interconnected (e.g., via connected stairwells), a two-dimensional (“2D”) floor plan of a building linked to a three-dimensional (“3D”) rendered floor plan of the building or otherwise associated therewith, etc. Additionally, although not described in Figure 1A , in some embodiments, the client computing device 175 (or other device, not shown) may receive and use information regarding the generated objects and / or rooms and / or building assessments in an additional manner (optionally in combination with the generated floor plan and / or other generated mapping-related information), such as to control or assist the automated navigation activities of these devices (e.g., autonomous vehicles or other devices), whether in lieu of or in addition to the display of the generated information.

[0042] In Figure 1A the depicted computing environment, the network 170 may be one or more publicly accessible linked networks that may be operated by various different parties, such as the Internet. In other embodiments, the network 170 may have other forms. For example, the network 170 may instead be a private network, such as a company or university network that is not fully or partially accessible to non-privileged users. In still other embodiments, the network 170 may include both private and public networks, where one or more of the private networks may access one or more of the public networks and / or be accessed from one or more of the public networks. Additionally, in various scenarios, the network 170 may include various types of wired and / or wireless networks. Additionally, the client computing devices 105 and 175 and the server computing system 180 may include various hardware components and stored information, as discussed in more detail below with respect to Figure 3 more detail.

[0043] In Figure 1AIn an example, the ICA system 160 can perform automated operations that involve generating a plurality of target panoramic images (e.g., each being a 360-degree panorama about a vertical axis) at a plurality of associated acquisition locations (e.g., within multiple rooms or other locations within a building or other structure and optionally around some or all of the exterior of the building or other structure), such as for generating and providing a representation of the interior of a building or other structure. In some embodiments, other automated operations of the ICA system can further include: analyzing information to determine the relative position / orientation between each of two or more acquisition locations; creating inter-panoramic position / orientation links in the panorama to each of one or more other panoramas based on such determined position / orientation; and then, providing information to display or otherwise present the plurality of linked panoramic images of the respective acquisition locations within the building, while in other embodiments, some or all of such other automated operations can instead be performed by the MIGM system.

[0044] Figure 1B A block diagram depicts an exemplary building interior environment in which linked panoramic images have been generated and are ready for generating and providing a corresponding building floor plan and for presenting the linked panoramic images to a user. In particular, Figure 1BIncluding a building 198 (in this example, a house 198), the building having an interior captured at least in part via a plurality of panoramic images, such as by a user (not shown) carrying a mobile device 185 with image acquisition capabilities and / or one or more separate camera devices 186 moving through the interior of the building to a series of multiple acquisition positions 210. Embodiments of the ICA system (e.g., the ICA system 160 on one or more server computing systems 180; some or all copies of the ICA system executed on the user's mobile device, such as the ICA application system 154 executed in the memory 152 of the device 185, etc.) can automatically perform or assist in capturing data representative of the interior of the building, and in some embodiments, further analyze the captured data to generate linked panoramic images, thereby providing a visual representation of the interior of the building. Although the user's mobile device may include various hardware components, such as one or more cameras or other imaging systems 135, one or more sensors 148 (e.g., gyroscope 148a, accelerometer 148b, compass 148c, etc., such as part of one or more IMUs or inertial measurement units of the mobile device); one or more hardware processors 132, memory 152, display 142 (e.g., including a touch-sensitive display screen), optionally one or more depth sensors 136, optionally other hardware elements (e.g., altimeter; light detector; GPS receiver; additional memory or other storage devices, whether volatile or non-volatile; microphone; one or more external lights; transmission capabilities to interact with other devices via one or more networks 170 and / or via direct device-to-device communication, such as interacting with an associated camera device 186 or a remote server computing system 180; microphone; one or more external lights, etc.), in at least some embodiments, the mobile device cannot access or use a device (such as a depth sensor 136) to measure the depth of an object in the building relative to the position of the mobile computing device, such that the relationship between different panoramic images and their acquisition positions can be determined at least in part or completely based on matching features in different images and / or by using information from other listed hardware components, but without using any data from any such depth sensor. Although not described for the sake of brevity, one or more camera devices 186 may similarly each include at least one or more image sensors and storage for storing the acquired target images, as well as transmission capabilities for transmitting the acquired target images to other devices (e.g., an associated mobile computing device 185, a remote server computing system 180, etc.), optionally one or more lenses and lights, and optionally in some embodiments, some or all of the other components of the mobile computing device are shown.Additionally, although the direction indicator 109 is provided for the viewer's reference, in at least some embodiments, the mobile device and / or the ICA system may not use such absolute direction information. Instead, for example, the relative directions and distances between panoramic images 210 may be determined without considering the actual geographical location or orientation.

[0045] In operation, the mobile computing device 185 and / or the camera device 186 (hereinafter referred to as "one or more image acquisition devices" for the Figure 1B example) reach a first acquisition position 210A within a first room inside a building (in this example, via the entrance from the outer door 190-1 to the living room), and capture visual data of a portion of the interior of the building visible from this acquisition position 210A (e.g., some or all of the first room, and optionally small portions of one or more other adjacent or nearby rooms, such as through doorways, halls, stairways, or other connecting passageways starting from the first room). In at least some cases, one or more image acquisition devices may be carried by one or more users or otherwise accompanied, while in other embodiments and cases, they may be mounted on or carried by one or more self-powered devices that move through the building on their own power. Additionally, in various embodiments, capturing visual data from the acquisition position may be performed in various ways (e.g., using one or more lenses that simultaneously capture all the image data, the associated user spinning his or her body while keeping one or more image acquisition devices fixed relative to the user's body, an automated device on which one or more image acquisition devices are mounted or carried rotating the one or more image acquisition devices, etc.), and may include recording video at the acquisition position and / or taking a series of one or more images at the acquisition position, including capturing visual information depicting many elements or other objects (e.g., structural details) visible in the images (e.g., video frames) that can be captured from or near the acquisition position. In Figure 1B the example, such elements or other objects include various elements (or structural "wall elements") that are part of the walls of a room that is structurally a house, such as doorways 190 and 197 and their doors (e.g., with revolving doors and / or sliding doors), windows 196, wall-to-wall boundaries (e.g., corners or edges) 195 (including corner 195-1 in the northwest corner of the house 198, corner 195-2 in the northeast corner of the first room (living room), and corner 195-3 in the southwest corner of the first room). Additionally, in Figure 1BIn the example of, such an element or other object may further include other elements within the room, such as furniture 191 to 193 (e.g., sofa 191; chair 192; table 193, etc.), pictures or paintings or a television or other objects 194 (such as 194-1 and 194-2) hanging on the wall, lighting fixtures, etc. One or more image acquisition devices further optionally capture additional data (e.g., additional visual data using the imaging system 135, additional motion data using the sensor module 148, additional depth data optionally using the distance measurement sensor 136, etc.) at or near the acquisition position 210A when rotating, and further optionally capture such additional data when the one or more image acquisition devices move to and / or from the acquisition position. In some embodiments, the actions of the one or more image acquisition devices may be controlled or facilitated via the use of one or more programs executed on the mobile computing device 185 (e.g., via automated instructions to the one or more image acquisition devices or to another mobile device (not shown) that carries those devices by its own power through the building; via instructions to the associated user in the room, etc.), such as the ICA application system 154 and / or the optional browser 162, the control system 147 for managing the I / O (input / output) and / or communication and / or network of the device 185 (e.g., receiving instructions from the user and presenting information to the user), etc. The user may also optionally provide a text or auditory identifier associated with the acquisition position, such as "entrance" for the acquisition position 210A or "living room" for the acquisition position 210B, while in other embodiments, the ICA system may automatically generate such identifiers (e.g., by automatically analyzing video and / or other recorded information of the building to perform corresponding automated determinations, such as by using machine learning), or may not use identifiers.

[0046] After the first acquisition location 210A has been suitably captured, one or more image acquisition devices (and the user, if present) may proceed to the next acquisition location (such as acquisition location 210B along the travel path 115), optionally recording movement data during movement between acquisition locations by one or more image acquisition devices, such as visual data and / or other non-visual data from hardware components (e.g., from one or more IMUs 148, from the imaging system 135 and / or one or more camera devices 186, from the distance measurement sensor 136, etc.). At the next acquisition location, one or more image acquisition devices may similarly capture one or more target images from that acquisition location and optionally capture additional data at or near that acquisition location. This process may be repeated for some or all of the rooms of the building and optionally outside the building, as shown for acquisition locations 210C through 210S. The video and / or other images acquired by one or more image acquisition devices for each acquisition location are further analyzed to generate a target panoramic image for each of acquisition locations 210A through 210S, including in some embodiments stitching together multiple component images to create the panoramic image and / or matching objects and other elements in different images.

[0047] In addition to generating such panoramic images, in at least some embodiments, further analysis may be performed by the MIGM system (e.g., concurrently with or after the image capture activity) to determine the room shape of each room (and optionally for other defined areas, such as a platform or other patio outside the building or other externally defined areas), including optionally determining the acquisition location information of each target image and optionally further determining the floor plan of the building and / or other relevant mapping information of the building (e.g., a set of interconnected linked panoramic images, etc.), e.g., in order to "link" together at least some of the panoramas and their acquisition locations (where for illustrative purposes some corresponding direction lines 215 between example acquisition locations 210A through 210C are shown), a copy of the MIGM system may determine the relative position information between pairs of acquisition locations that are visible to each other, store the corresponding inter-panoramic links (e.g., links 215-AB, 215-BC, and 215-AC between acquisition locations 210A and 210B, 210B and 210C, and 210A and 210C, respectively), and in some embodiments and scenarios, further link at least one acquisition location that is not visible to each other (e.g., link 215-BE (not shown) between acquisition locations 210B and 210E; link 215-CS (not shown) between acquisition locations 210C and 210S, etc.).

[0048] Additionally, the mobile computing device 185 and / or the camera device 186 may operate under the control of the BUAM system (whether it is the system 140 on one or more server computing systems 180 or the BUAM application 156 executed in the memory 152 of the mobile computing device 185) to capture images of the room and the objects in the room and their target attributes, either instead of or in addition to performing the image acquisition operations of the ICA system (e.g., in some embodiments, capturing images for both systems simultaneously, capturing images only for the BUAM system and not for the ICA system, etc.). In a manner similar to that discussed above with respect to the ICA system, the image acquisition device may move through some or all of the rooms of the building 198 to capture initial images and additional images (e.g., simultaneously, such as if the analysis of the visual data of the initial images is performed in real-time or near real-time, such as within seconds or minutes of acquiring the initial images; in two or more different trips through the building, such as one or more first trips for capturing the initial images and one or more second trips for capturing the additional images, etc.), but in other cases, the BUAM system may only acquire additional images (e.g., if images from another system (such as the ICA system) are used as the initial images) and / or only acquire initial images (e.g., if the initial images include sufficient visual details about all the objects and the target attributes of the objects to perform the assessment of the target attributes and the evaluation of the objects and the room). The acquisition of the initial images and / or additional images by the BUAM system may, for example, include passing through the acquisition location 210 either entirely or partially along the path 115, and optionally may include deviating from the path to capture sufficient details about individual objects and / or object attributes. In at least some cases, one or more image acquisition devices may be carried or otherwise accompanied by one or more users when participating in capturing the initial images and / or additional images for the BUAM system, while in other embodiments and cases, they may be mounted on or carried by one or more self-powered devices that move through the building on their own power. Additionally, in various embodiments, the capture of the visual data may be performed in various ways, as discussed in more detail above with respect to the operation of the ICA system. One or more image acquisition devices further acquire additional data for the BUAM system (e.g., additional visual data using the imaging system 135, additional motion data using the sensor module 148, optionally additional depth data using the distance measurement sensor 136, etc.), as well as data input or otherwise provided by one or more accompanying users (e.g., the BUAM system operator user).In some embodiments, the actions of one or more image acquisition devices may be controlled or facilitated via the use of one or more programs executing on the mobile computing device 185 (e.g., via automated instructions to one or more image acquisition devices or to another mobile device (not shown) that carries those devices through the building under its own power; via instructions to associated users in the room, etc.), such as the BUAM application system 156 and / or the BUAM system 140. After capturing the various initial and additional images, as well as any other additional data, the BUAM system continues to perform its automated operations to assess target attributes and evaluate usability information for the object, room, and / or building, and uses the generated usability information in various ways.

[0049] about Figures 1A to 1B Various details are provided but it is understood that the details provided are non-exclusive examples included for illustrative purposes and that other embodiments may be otherwise performed without some or all of such details.

[0050] Figures 2A to 2X Examples are described of automatically capturing images associated with a building and analyzing visual data of the images (and optionally additional types of captured data) to generate and use various information about the building and its rooms, such as generating floor plans of the building, room shapes of the rooms of the building, rooms and objects in the rooms (and assessments of target attributes of the objects), and usability assessments of the building, at least some of the images being in a Figure 1B Captured at acquisition location 210 within building 198 discussed in the illustration.

[0051] In particular, Figure 2A An example image 250a is shown, such as in Figure 1B A non-panoramic stereoscopic image (or a northeast-facing subset view of a 360-degree panoramic image captured from the capture location and formatted in a rectilinear manner) of a living room of a house 198 captured from a capture location 210B in a northeast direction, such as captured by an ICA system and / or by a BUAM system as an initial image. A direction indicator 109a is further displayed in this example to illustrate the northeast direction in which the image was captured. In the example shown, the displayed image includes built-in elements (e.g., lighting fixtures 130a), furniture (e.g., chairs 192-1), two windows 196-1, and a picture 194-1 hanging on the north wall of the living room. Inter-room passages (e.g., doorways or other wall openings) leading into and out of the living room are not visible in this image. However, multiple room boundaries are visible in image 250a, including a horizontal boundary between a visible portion of the north wall of the living room and the ceiling and floor of the living room, a horizontal boundary between a visible portion of the east wall of the living room and the ceiling and floor of the living room, and an inter-wall vertical boundary 195-2 between the north wall and the east wall.

[0052] Figure 2B Continuing to show Figure 2A an example, and illustrates additional stereo image 250b (or a northwestern subset view of a 360-degree panoramic image captured from the acquisition position 210B and formatted in a straight line) captured by one or more image acquisition devices in the living room of house 198 at Figure 1B from the acquisition position 210B in the northwestern direction. In this example, direction indicator 109b is further shown to illustrate the northwestern direction of the captured image. In this example image, a small portion of one of the windows 196-1, together with a portion of window 196-2 and new lighting fixture 130b, continues to be visible. Additionally, the horizontal and vertical room boundaries are visible in image 250b in a manner similar to that of Figure 2A ...

[0053] Figure 2C Continuing to show Figures 2A to 2B an example, and shows a third stereo image 250b (or a southwestern subset view of a 360-degree panoramic image captured from the acquisition position 210B and formatted in a straight line) captured by one or more image acquisition devices in the living room of house 198, such as in the southwestern direction from the acquisition position 210B. In this example, direction indicator 109c is further shown to illustrate the southwestern direction of the captured image. In this example image, a portion of window 196-2 continues to be visible, as well as the sofa 191 and the visual horizontal and vertical room boundaries in a manner similar to that of Figure 1B ... Figure 2A and Figure 2B ... Figure 1B which is identified as the door 190-1 leading to the outside of the house). It should be understood that multiple other stereo images can be captured from the acquisition position 210B and / or other acquisition positions and shown in a similar manner.

[0054] Figure 2D Shows Figure 1B other information 255d of a portion of house 198, including the living room and a limited portion of other rooms to the east of the living room. As regarding Figure 1B and Figures 2A to 2CDiscussion, in some embodiments, target panoramic images may be captured at various locations within a house, such as locations 210A and 210B in a living room, and the corresponding visual content of one or both of such obtained target panoramic images is subsequently used to determine the room shape of the living room. Additionally, in at least some embodiments, additional images may be captured, such as in the case where one or more image capture devices (not shown) capture video or one or more other series of continuous or nearly continuous images while moving through the interior of the house. In this example, information about a portion of the path 115 shown in Figure 1B is shown, and in particular, a series of locations 215 along the path where one or more video frame images (or other series of continuous or nearly continuous images) of the surrounding interior of the house may be captured as the one or more image capture devices move (e.g., if capturing video data). Examples of such locations include capture locations 240a to 240c, where other information is related to the video frame images captured from Figures 2E to 2J the locations shown. In this example, the locations 215 along the path are shown as being separated by a short distance (e.g., one foot, one inch, a fraction of an inch, etc.), but it should be understood that the video capture may be substantially continuous. Thus, in at least some embodiments, only a subset of such captured video frame images (or other images from a series of continuous or nearly continuous images) may be selected and used for further analysis, such as images separated by a defined distance and / or a defined amount of time between captures (e.g., one second, a fraction of a second, several seconds, etc.) and / or based on other criteria.

[0055] Figures 2E to 2J Continuing with the example shown in Figures 2A to 2D and showing additional information regarding the living room and regarding a portion (such as determined by the MIGM system) of an estimate of a possible shape of the room as a type of analysis of 360° image frames of video captured along path 155. Although not illustrated in these figures, similar techniques may be performed on target panoramic images captured by a camera device at two or more of the acquisition locations 210A, 210B, and 210C, whether to supplement the analysis Figure 2D of the additional image frames shown (e.g., to generate additional estimates of the possible shape of the room using the visual data of the target images) or instead of analyzing Figure 2D the additional image frames shown. In particular, Figure 2E includes information 255e, which states that the 360° image frame taken from location 240b will share information about various visible 2D features with the 360° image frame taken from location 240a, but for simplicity, only a limited subset of such features of a portion of the living room is shown in Figure 2E . In Figure 2EIn [the figure], an example line of sight 228 from position 240b to various example features in the room is shown, and a similar example line of sight 227 from position 240a to the corresponding features is shown, which shows the degree of difference between the perspectives at significantly separated capture positions. Thus, analyzing a series of images corresponding to Figure 2D position 215 using SLAM and / or MVS and / or SfM techniques can provide various information about the features of the living room, such as about Figures 2F to 2I as further described.

[0056] In particular, Figure 2F information 255f of the northeast part of the living room visible in a subset of 360° image frames taken from positions 240a and 240b is shown, and Figure 2G information 255g of the northwest part of the living room visible in other subsets of 360° image frames taken from positions 240a and 240b is shown, where various example features in those parts of the living room are visible in the two 360° image frames (e.g., corners 195-1 and 195-2, windows 196-1 and 196-2, etc.). As part of the automated analysis of 360° image frames using SLAM and / or MVS and / or SfM techniques, partial information about planes 286e and 286f corresponding to parts of the north wall of the living room can be determined based on the detected features, and partial information 287e and 285f about parts of the east and west walls of the living room can be similarly determined based on the corresponding features identified in the images. In addition to this partial plane information identifying the detected features (e.g., each point in the determined sparse 3D point cloud from image analysis), SLAM and / or MVS and / or SfM techniques can also determine information about: the possible positions and orientations / directions 220 of the image subset from capture position 240a and the possible positions and orientations / directions 222 of the image subset from capture position 240b (e.g., positions 220g and 222g in Figure 2F and, optionally Figure 2F the directions 220e and 222e of the image subsets shown; and positions 220g and 222g in Figure 2G corresponding to capture positions 240a and 240b, respectively, and, optionally Figure 2G the directions 220f and 222f of the image subsets shown). Although Figure 2F and Figure 2GOnly the features of a portion of the living room are shown, but it will be understood that other portions of the 360° image frames corresponding to other parts of the living room can be analyzed in a similar manner to determine possible information about the various walls of the room and other features (not shown) of the living room. Additionally, a similar analysis can be performed between some or all of the other images at the selected location 215 in the living room, resulting in multiple determined feature planes corresponding to portions of the walls of the room from various image analyses.

[0057] Figure 2H Continuing to show Figures 2A to 2G an example, and showing information 255h of multiple determined feature planes corresponding to portions of the west and north walls of the living room from the analysis of 360° image frames captured at positions 240a and 240b. The shown plane information includes a determined plane 286g near or at the north wall (and thus the corresponding possible positions of portions of the north wall), and a determined plane 285g near or at the west wall (and thus the corresponding possible positions of portions of the west wall). As would be expected, there are many variations in the different determined planes of the north and west walls from different features detected in the analysis of the two 360° image frames, such as differences in position, angle, and / or length, as well as missing data for some portions of the walls, resulting in uncertainty about the actual exact positions and angles of each wall. Although Figure 2H not illustrated in [reference], it should be understood that similar determined feature planes of the other walls of the living room will be detected similarly, as well as determined feature planes corresponding to features (e.g., furniture) not along the walls.

[0058] Figure 2I Continuing to show Figures 2A to 2H an example, and showing information 255i of additional determined feature plane information corresponding to portions of the west and north walls of the living room from the analysis of various additional 360° image frames selected from additional positions 215 along path 115 in the living room. As would be expected, in this example, the analysis of the other images provides even greater variations in the different determined planes of the north and west walls. Figure 2I Also shown is additional determined information used to aggregate information about various portions of the determined feature planes to identify possible partial positions 295a and 295b of the west and north walls, as shown by Figure 2J information 255j. In particular, Figure 2IShows information 291a about the normal orthogonal directions of some of the determined feature planes corresponding to the west wall, and additional information 288a about those determined feature planes. In an example embodiment, the determined feature planes are clustered to represent the assumed wall positions of the west wall, and the information about the assumed wall positions is combined to determine possible wall positions 295a, such as weighting information from various clusters and / or basic determined feature planes. In at least some embodiments, the assumed wall positions and / or normal information are analyzed via the use of machine learning techniques to optionally further apply assumptions or other constraints (such as 90° corners, as shown by the information 289 of Figure 2H and / or having flat walls) or apply the results of the analysis to determine the resulting possible wall positions. A similar analysis can be performed for the north wall using information 288b about the corresponding determined feature planes and additional information 291b about the resulting normal orthogonal directions of at least some of those determined feature planes. Figure 2J Shows the resulting possible partial wall positions 295a and 295b of the west wall and north wall of the living room respectively, including optionally estimating the positions of missing data (e.g., via interpolation and / or extrapolation using other data).

[0059] Although Figure 2I not shown in, it should be understood that similar determined feature planes and corresponding standard directions of the other walls of the living room will be similarly detected and analyzed to determine their possible positions, resulting in an estimated partial room shape of the living room based on the visual data collected in the living room by one or more image acquisition devices. Additionally, a similar analysis is performed for each room of the building, thereby providing an estimated partial room shape for each room. Additionally, although Figures 2D to 2J not described in, in some embodiments, the analysis of the visual data captured by one or more image acquisition devices in the living room can be supplemented and / or replaced by the analysis of depth data (not shown) captured by one or more image acquisition devices in the living room, such as to directly generate an estimated 3D point cloud from the depth data representing the walls and optionally the ceiling and / or floor of the living room. Although Figures 2D to 2J also not described in, in at least some embodiments, other room shape estimation operations can be performed using only a single target panoramic image, such as the analysis of the visual data of the target panoramic image via one or more trained neural networks, as described in more detail elsewhere in this document.

[0060] Figure 2K Continues to show Figures 2A to 2JExample, and shows information 255k about additional information that can be generated from one or more images in a room and used in one or more ways in at least some embodiments. In particular, an image (e.g., a video frame) captured in the living room of house 198 can be analyzed to determine an estimated 3D shape of the living room, such as determined from a 3D point cloud of features detected in the video frame (e.g., using SLAM and / or SfM and / or MVS techniques, and optionally further based on IMU data captured by one or more image acquisition devices). In this example, information 255k reflects an example portion of such a point cloud of the living room, such as in this example, corresponding to the northwest direction of the living room (e.g., including the northwest corner 195-1 of the living room and window 196-1) in a manner similar to image 250c of Figure 2C The area 299 corresponding to window 196-1 and the boundary 298 corresponding to the north wall of the living room can be identified. It should be understood that in other embodiments, such an estimated 3D shape of the living room can be determined by using depth data captured in the living room by one or more image acquisition devices, either supplementing or replacing the visual data of one or more images captured in the living room by one or more image acquisition devices. Additionally, it should be understood that various other walls and other features can be similarly identified in the living room and other rooms of house 198.

[0061] Figure 2L Shows additional information 255l, which corresponds to after determining the final estimated room shape of the rooms on the shown floor of house 198 (e.g., the 2D room shape 236 of the living room), in this example at least partially based on the matching room shape information of the connecting inter-room channels and adjacent rooms between the rooms, positioning the estimated room shapes of the rooms relative to each other. In at least some embodiments, such information can be considered a constraint on the positioning of the rooms, and determining the best or otherwise preferred solution to those constraints. Figure 2L Examples of such constraints in Figures 2E to 2J and / or Figures 2P to 2XMatch 231 the detected channels in the automated image analysis being discussed so that the positions of those channels are co-located, and match 232 the shapes of adjacent rooms to connect those shapes (e.g., as shown for rooms 229d and 229e and rooms 229a and 229b). In other embodiments, various other types of information may be used for room shape positioning, either in addition to or instead of channel-based constraints and / or room shape-based constraints, such as exact or approximate dimensions of the overall size of the house (e.g., based on additional metadata available about the building, analysis of images from one or more image acquisition locations outside the building, etc.). External house information 233 may further be identified and used as a constraint (e.g., at least in part based on the automated identification of channels and other features corresponding to the outside of the building, such as windows), such as to prevent another room from being placed in a location that has been identified as outside the building. In Figure 2L In an example, the final estimated room shape used may be a 2D room shape, or instead a 2D version of the 3D final estimated room shape may be generated and used (e.g., by taking a horizontal slice of the 3D room shape).

[0062] Figure 2M Continuing with FIG. 2 - O shows an Figures 2A to 2L example, and shows the surveying information that may be generated from the Figures 2A to 2L and Figures 2P to 2V types of analysis discussed, such as generated by the MIGM system. In particular, Figure 2M shows an example 2D floor plan 230m that may be constructed based on the positioning of the determined final estimated room shape, in this example the floor plan including indications of walls as well as doorways and windows. In some embodiments, such a floor plan may show other information, such as other features automatically detected by the analysis operations and / or subsequently added by one or more users. For example, Figure 2NShows a modified floor plan 230n including various types of additional information, such as information that can be automatically identified and added to the floor plan 230m from the analysis of visual data from an image and / or from depth data, including one or more of the following types of information: room labels (e.g., "Living Room" for the living room), room dimensions, visual indications of furniture or appliances or other fixed features, visual indications of the location of additional types of associated and linked information (e.g., panoramic images and / or stereoscopic images collected at a specified collection location that the end user can select to further display; audio annotations and / or sound recordings, etc. that the end user can select to further present), visual indications of doorways and windows. In other embodiments and scenarios, some or all of this type of information may instead be provided by one or more MIGM system operator users and / or ICA system operator users. Additionally, if evaluations and / or other information generated by the BUAM system are available, they can be similarly added to the floor plan 230m and / or 230n or otherwise associated with the floor plan, whether supplementing or replacing some or all of the other additional types of information shown for the floor plan 230n relative to the floor plan 230b. Additionally, when displaying the floor plan 230m and / or 230n to the end user, one or more user-selectable controls can be added to provide interactive functionality as part of the GUI (Graphical User Interface) screen 255n, such as indicating the current floor to be displayed, allowing the end user to select a different floor to display, etc. In this example, the corresponding example user-selectable control 228 is added to the GUI. Additionally, in some embodiments, floor or other building changes can also be made directly from the displayed floor plan, such as via selecting a corresponding connection passage (e.g., a staircase leading to a different floor), and other visual changes can be made directly from the displayed floor plan by selecting the corresponding displayed user-selectable control (e.g., a control corresponding to a specific image at a specific location and receiving the display of that image, whether replacing or supplementing the previous display of the floor plan from which the image was selected). In other embodiments, information for some or all different floors can be displayed simultaneously, such as in the form of separate sub-floor plans for individual floors or alternatively by integrating the room connection information for all rooms and floors into a single floor plan that is displayed immediately and simultaneously. It should be understood that various other types of information can be added in some embodiments, some of the types of information shown may not be provided in some embodiments, and in other embodiments, the visual indications of linked and associated information and user selections thereof can be displayed and selected in other ways.

[0063] Figure 2 - O continues to show Figures 2A to 2N an example of, and shows what can be disclosed and displayed herein (e.g., in a manner similar to Figure 2NAdditional information 265o generated by automated analysis techniques in the GUI), which in this example is a 2.5D or 3D model floor plan of a house. Such a model 265o can be additional surveying-related information generated based on floor plans 230m and / or 230n, which shows additional information about height to illustrate the visual position of features such as windows and doors in the walls, or alternatively a combined final estimated room shape, which is a 3D shape. Although not illustrated in FIG. 2-O, in some embodiments, the additional information can be added to the displayed walls, such as from images taken during video capture (e.g., rendering and illustrating actual paintings, wallpapers, or other surfaces from the house on the rendered model 265), and / or can be used in other ways to add specified colors, textures, or other visual information to the walls and / or other surfaces. Additionally, Figure 2N Some or all of the additional types of information shown in can be similarly added to and shown in the floor plan model 265o.

[0064] Figures 2P to 2X Continuing to show Figure 2A the example of FIG. 2-O, where Figure 2P further shows showing Figure 1B information 255p of a part of the living room of house 198. In particular, in Figure 2P the example, an image 250p of the southwestern part of the living room is shown (in a manner similar to Figure 2C image 250c), but with additional information superimposed on the image to illustrate information determined about the objects and target attributes identified in that part of the room for further analysis, as well as information about the positions of those objects. In particular, in this example, the west window (element 196-2 of image 250c) is selected as the object of interest in the room, for which a corresponding "west window" label 246p2 has been determined (either automatically or at least in part based on information provided by one or more associated users), and the automatically determined position 199b of the object in the image is shown (in this example, the position is the bounding box of the object). Figure 2P Information 255p further shows a list 248p of objects and target attributes of interest identified at least in part based on the visual data of image 250p, which indicates that the target attributes of interest of the west window include its size and information about the view through the window. Image 250p further shows the door ( Figure 2CElement 190-1) has been identified as an object of interest, where the "front door" label 246p1 is shown (either automatically or at least in part based on information provided by one or more associated users) and the automatically determined bounding box position 199a. Additionally, information 248p indicates that the target attributes of the door include a door handle and door hinge that are further visually indicated as 131p on image 250p. Additionally, image 250p also shows a couch ( Figure 2C Element 191) has been identified as an object of interest, where an automatically determined pixel-level mask position 199c is identified for the couch, but no label is shown in this example. Other objects can be similarly identified, such as one or more ceiling light fixtures indicated in information 248p, but not shown in the example image 250p (e.g., at least in part based on room expectations for a typical "living room" or a list of defined types of typical objects). Similarly, other target attributes can be identified, such as the latch hardware for the west window indicated in information 248p, but not shown in the example image 250p (e.g., at least in part based on object expectations for a type "window" or a list of defined types of typical target attributes). Additionally, the "living room" label 246p3 for the room has been determined (either automatically or at least in part based on information provided by one or more associated users) and is shown. In some embodiments, such information 250p and / or information 248p can be displayed, for example, to an associated user in the room (e.g., on the user's mobile computing device or the user's other image capture device), as part of specifying to the user additional data to capture, such as to identify specific objects and / or target attributes and their locations. Figure 2Q An alternative image 250q is provided as part of information 255q. In this example, the alternative image is a panoramic image of a 360° visual coverage of the living room. Such a panoramic image can be used instead of or in addition to a stereoscopic image such as image 250p to identify objects and target attributes and additional relevant information (e.g., location, label, etc.), as well as to evaluate the overall layout of the objects in the room and / or the expected traffic flow in the room. The example panoramic image 250q similarly shows the position bounding boxes 199a and 199b for the front door and west window objects (in this example, instead of the couch object), as well as an additional position bounding box 199d for the ceiling light 130b. It should be understood that in other embodiments, various other types of objects and / or target attributes can be identified.

[0065] Figures 2R to 2T Continuing to show Figures 2P to 2Q the example, and further showing information regarding instructions that can be provided to indicate additional data to capture in the living room and the corresponding additional data captured. In particular, Figure 2Rshows an image 250r that can be displayed to an associated user in a living room, such as in a manner similar to Figure 2P image 250p, but in which additional information 249r provides instructions to the associated user for obtaining additional data regarding the front door object (including target attributes regarding hinges and door handles), as well as options for the user to receive examples and optionally additional indication information. Although Figure 2R not illustrated, similar instructions can be provided for other objects such as the west window and / or the sofa and / or the ceiling light, such as provided continuously after instruction 249r has been provided and the corresponding additional data has been obtained, or provided concurrently with instruction 249r. Figure 2S shows an additional image 250s representing additional data regarding the front door captured in the living room (e.g., in response to instructions provided by the BUAM system), such as having additional details not available in the visual data of image 250r regarding the door. Additionally, image 250a is overlaid with examples of additional instructions or other information that can be provided to the associated user (e.g., before or after the user captures an image of the front door shown in image 250a), such as an instruction 249s1 indicating to recapture a better-lit image after capturing image 250s and / or a notification that one or more visible objects before or after capturing image 250s do not actually appear to be the front door object shown in image 250r (e.g., based on an automated comparison of the visual data in the two images). Figure 2S Further shows an example 249s2 of additional instructions regarding additional non-visual data to be captured regarding the front door object (whether before or after capturing image 250s), such as providing a short text description of the door material and age and / or recording and providing a short video including the view through the open door of the door being opened. Figure 2T Further provides example additional images 250t1 and 250t3 captured to provide additional details of the identified target attributes regarding the front door object, where image 250t1 shows additional details regarding the door handle and image 250t3 shows additional details regarding one of the hinges. In this example, image 250t1 is further overlaid with an example instruction 249t1 indicating that insufficient details regarding the door handle were obtained in image 250t1 (e.g., because the image did not sufficiently focus only on the door handle) and that a new alternative additional image should be captured, where image 250t2 provides an example of such an alternative additional image to be used in place of image 250t1. Image 250t3 further provides an example of additional instruction or other notification information 249t2 for obtaining additional data regarding the visible hinge, in this example, for the associated user to confirm that the visual data in image 250t3 is for the front door object in the living room. It should be understood that in Figures 2P to 2TThe types of instructions described and the manner in which they are provided to the associated user are non-exclusive examples provided for illustrative purposes, and in other embodiments, similar and / or other types of information may be provided in other ways.

[0066] Figure 2U Continuing, examples are shown Figures 2P to 2T and examples of additional data regarding the living room that may be obtained at least in part based on an analysis of one or more initial room-level images of the living room are provided, such as the panoramic image 250q and / or a plurality of stereograms including images 250a to 250c and including all or substantially all of the visual data of the living room. In particular, Figure 2U information 255u is shown, which shows alternative examples 237a and 237b of the room shape of the living room (e.g., as may be determined by the MIGM system, as discussed in more detail elsewhere herein), and additional data 236u and 238 of the room shape 237a that may be determined at least in part based on the automated operation of the BUAM system and optionally additional actions of the associated user. In this example, the information 236u shown provides an example of the expected communication traffic information of the living room, such as at least in part based on the determined layout of the living room (not shown) (e.g., using information about the furniture and wall openings in the living room). Additionally, the information 238 shown indicates that in this example, the target attribute of the west window may have been rated as showing a mountain view (e.g., at least in part based on an automated determination using the visual data visible through the window; at least in part using information from the associated user; at least in part using information from other sources, such as publicly available data, etc.). It should be understood that Figure 2U these types of additional information described are non-exclusive examples provided for illustrative purposes, and in other embodiments, similar and / or other types of information may be determined in other ways.

[0067] Figures 2V to 2W Continuing, examples are shown Figures 2P to 2U and examples of additional data regarding those other rooms that may be obtained at least in part based on an analysis of one or more initial room-level images of other rooms of the building are provided. In particular, Figure 2V information 255v including image 250v is shown, such as for Figure 1B the bathroom 1 of the exemplary house 198 shown in Figure 2N and identified in Figure 2PIn the manner of Image 250p, Image 250v includes indications 131v of objects and / or target attributes identified in the bathroom for capturing additional data, which in this example includes tile floors, sink countertops, sink faucets and / or other sink hardware, bathtub faucets and / or other bathtub hardware, toilets, etc. However, position information, labels, and provided instructions are not shown in this example. In a similar manner, Figure 2W Information 255w is shown including Image 250w, such as for Figure 1B the kitchen of the exemplary house 198 shown (as identified in Figure 2N and Figure 2-O). In a manner similar to Figure 2V Image 250v, Image 250w includes indications 131w of objects and / or target attributes identified in the kitchen for capturing additional data, which in this example includes refrigerators, stoves on the kitchen island, sink faucets and / or other sink hardware, countertops next to the sink and / or backsplashes, etc. However, position information, labels, and provided instructions are not shown in this example. It should be understood that various types of corresponding instructions can be generated and provided to obtain additional data regarding such identified objects and / or target attributes, and Figures 2V to 2W such types of additional data are shown in

[0068] Figure 2X as non-exclusive examples provided for illustrative purposes, such that in other embodiments, similar and / or other types of information can be determined in other ways. Figures 2P to 2W Figure 1B Figure 2U Figure 2N Figure 2X ​​​​The additional data types shown are non-exclusive examples provided for illustrative purposes, such that in other embodiments, similar and / or other types of information may be determined in other ways.

[0069] Various details have been provided with respect to Figures 2A to 2X but it should be understood that the details provided are non-exclusive examples included for illustrative purposes and that other embodiments may be carried out in other ways without some or all of such details.

[0070] As a non-exclusive example embodiment, the automated operation of the BUAM system may include next actions regarding providing instructions related to capturing additional data for evaluating the usability of an object, a room, and a building. In this example embodiment, non-exclusive examples of evaluating an object of interest and rating target attributes may include going back to questions such as the following: Are the kitchen cabinets new? Are they up to the ceiling? What kind of bathroom fixtures are there? What is the condition of the door and window frames? What is the condition of the gutters and downspouts? What kind of pipes are under the sink? What kind of hot water tank is there? To this end, the BUAM system of the example embodiment may generate and provide instructions and related information such as the following non-exclusive examples: "Take a photo of the kitchen sink", "Zoom in some so we can see more details", "Are you sure that's the sink?", "Thanks for the photo of the bathtub. Is this from the master bedroom bathroom or the hall bathroom?", "Can you take a close-up of the drain pipe?", etc. As part of doing so, the BUAM system of the example embodiment may perform automated operations to classify or detect common house features such as sinks, drain pipes, door frames from images or videos, such as building convolutional neural network models for these, optionally along with a predefined checklist of target attributes (also referred to as "features" in this example embodiment) for which additional data is to be captured (e.g., for a bathtub, obtain and analyze a close-up image of the drain pipe; for a doorway, obtain and analyze a close-up image of the door jamb, etc.), including verifying that the drain pipe is visible in the corresponding captured additional image and that it is of a particular minimum size. As part of doing so, such a BUAM system may provide a GUI (or other user interface) that provides a list of the identified objects and / or target attributes for which additional data is to be captured to an associated user, as well as corresponding examples of good images of those types.

[0071] The BUAM system of the example embodiment may, for example, implement a workflow having the following steps to evaluate a room of a house:

[0072] 1) Start with an initial set of images from the house and room labels or object labels generated by a machine learning model and / or a user.

[0073] 2) Given this list of labeled rooms and / or objects, generate a list of target attributes to be captured or investigated.

[0074] 3) Use a detector model to determine whether the initial image already contains visual data of the target attributes at a sufficient image resolution.

[0075] 4) For target attributes lacking such visual data in the initial image, prompt the associated user to capture them, such as in the following ways:

[0076] a. Present to the user one or more initial images of the room of interest as a "constructive shot".

[0077] b. Optionally, show example images illustrating the details and camera angles to be captured.

[0078] c. Instruct the user to capture images and / or other media (e.g., video, 3D models, etc.) with visual data of the indicated target attributes and / or objects.

[0079] d. Analyze the captured media through automated on-board processing in order to:

[0080] i. Verify the presence of the desired data at the desired resolution.

[0081] ii. Determine other characteristics of the capture.

[0082] e. Optionally, verify that the background in the captured media matches the background in the "constructive shot" or other previously captured images of the room. This verification can be performed, for example, using image information (e.g., by analyzing the background) and / or using telemetry information (e.g., by checking that the camera pose information in the captured media is consistent with the camera pose information in the initial image).

[0083] f. If steps (d) or (e) reveal problems with the captured media, prompt the user to recapture to correct the problems.

[0084] g. Optionally, prompt the user to input more data about the target attributes and / or objects that cannot be visually determined.

[0085] Regarding step 1 above, the initial images can be panoramic images and / or stereoscopic images (e.g., submitted by the seller or agent or the person taking the pictures during the listing of the property) and are ideally captured separately in each room. They can be annotated with room classification labels at the time of submission (e.g., the user can label the images as "kitchen", "bedroom", "living room", etc.) and / or can be labeled after submission using a machine learning model for room classification. Additionally, there may be image regions or points where the user has added "point of interest" labels to objects (e.g., "industrial oven" or "new shower"), which can be further used to identify the objects of interest and / or the associated target attributes. Such operations can be performed, for example, on a mobile computing device used as an image acquisition device and / or on a remote server computing device.

[0086] Regarding step 2 above, the BUAM system can perform a mapping from the labels to the target attributes and / or additional data types to be captured. For example, the mapping can indicate information such as in the following non-exclusive examples: in the kitchen, a close-up of the stove (so that the viewer can distinguish the brand or inspect its controls); in the bedroom, close-ups of the hardware of each sink, bathtub hardware, all sides of the bathtub, etc.; if there is a fireplace, information on whether it burns gas or wood, etc.

[0087] Regarding step 3 above, the BUAM system can use one or more deep learning detector models to detect certain objects and / or target attributes in the images. For example, such detections can include one or more of the following non-exclusive examples: in the kitchen, detecting the stove, sink, and refrigerator; in the bedroom, detecting each sink; in the living room, detecting the fireplace or wood stove, etc. Such detector models can extract boundary regions from the input images to determine the object locations (e.g., a <width, height> pixel rectangle, whose upper left corner is the <x, y> of the sink, auxiliary boundary regions for target attributes such as sink hardware and / or drain pipes, etc.). The BUAM system can use predefined information specifying the minimum desired image size and area in pixels for the additional data to be captured for each type of detectable object and target attribute, and the BUAM system will then verify this in the visual data of the additional images captured (e.g., to see if they meet the desired size and area).

[0088] Regarding step 4a above, the BUAM system can present an initial image of the bathroom along with prompts such as "Please capture a photo / video / 3D model that can capture the sink hardware". Regarding step 4b above, for each type of object and target property, the BUAM system can have a library of definitions of standard example images. Regarding step 4c above, the BUAM system can use different types of media in different scenarios, such as images for obtaining fine details (and optionally capturing additional data, such as simultaneously taking a second image using the wide-angle lens of the image acquisition device, providing a narrow / wide view pair), short videos for evaluating functionality (e.g., a short video of a fully operational faucet to evaluate water pressure), 3D models for evaluating larger scenes (e.g., using the lidar scanner of a phone to capture the perimeter of a house), etc. Regarding step 4d above, the BUAM system can apply models similar to those in step 3 to detect objects and target properties, extract their location areas, and compare them with the desired size. Such operations can be performed, for example, on a mobile computing device used as the image acquisition device. Other verification operations can be performed with respect to: image brightness (e.g., if the image is captured in a dark space such as under a cabinet or near a stove / hot water heater), characteristics of 3D capture (e.g., does the captured 3D model of the house perimeter form a closed loop? - if not, provide instructions to capture the missing area), etc. Regarding step 4e above, the BUAM system can perform verification activities to ensure that the captured image is in the correct room (e.g., check that the scene background of the captured additional image of the sink is from the correct bathroom, optionally using information from the narrow / wide field of view pair (if available); use image-to-image feature matching to match the visual data in the captured additional image to the visual data of one or more in the initial room-level image; verify similar colors or textures between the captured additional image and the visual data of one or more in the initial room-level image), etc. Such operations can be performed, for example, on a mobile computing device used as the image acquisition device. Regarding step 4g above, the BUAM system can perform automated operations, such as providing prompts to enter the year of the last replacement of an object (e.g., hot water tank), specifying when the wooden floor was last renovated and / or whether renovation is needed, etc. Additionally, the automated operations of the BUAM system can include prioritizing the order of capturing additional images based on one or more defined criteria, such as capturing visual data and / or other data about kitchen appliances before capturing visual data and / or other data about kitchen drawer handles (e.g., if the kitchen appliance information has greater weight or has other impacts on the determination of the usability of the kitchen).

[0089] For efficiency purposes, the analysis of the visual data of the initial image and / or the additional images captured may include, if possible, using downsampling of the captured images (to reduce the resolution of the resulting images). For example, if data is available from a lidar sensor to give 3D geometric information, this may also help select an appropriate amount of downsampling to perform. Additionally, some or all of the operations described above for the example embodiments of the BUAM system may be performed on a mobile computing device used as an image capture device and / or may be performed on one or more remote server computing systems (e.g., if the operations cannot be performed efficiently or quickly enough on the mobile computing device). In the latter case, there may be a time delay between the capture of the media and the issuance of the relevant feedback, and if so, for all object and / or target attributes, the feedback in step 4f may be aggregated and presented together later.

[0090] Various details have been provided with respect to this example non-exclusive embodiment, but it should be understood that the details provided are included for illustrative purposes and that other embodiments may be otherwise implemented without some or all of such details.

[0091] Figure 3 is a block diagram of an embodiment showing one or more server computing systems 300 implementing an embodiment of the BUAM system 340, and one or more server computing systems 380 implementing embodiments of the ICA system 387 and the MIGM system 388. The one or more server computing systems and the BUAM system may be implemented using multiple hardware components that form an electronic circuit suitable for and configured to perform at least some of the techniques described herein in joint operation. In the embodiment shown, each server computing system 300 includes one or more hardware central processing units (“CPUs”) or other hardware processors 305, various input / output (“I / O”) components 310, a storage device 320, and a memory 330, where the I / O components shown include a display 311, a network connector 312, a computer-readable media drive 313, and other I / O devices 315 (e.g., a keyboard, a mouse, or other pointing device, a microphone, a speaker, a GPS receiver, etc.). Each server computing system 380 may include hardware components similar to those of the server computing system 300, including one or more hardware CPU processors 381, various I / O components 382, a storage device 385, and a memory 386, but some details of the server 300 are omitted in the server 380 for brevity.

[0092] One or more server computing systems 300 and the executing BUAM system 340 can communicate with other computing systems and devices via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.), such other computing systems and devices being, for example: user client computing devices 390 (e.g., for viewing floor plans, associated images, objects, and / or room and / or building assessments and / or other relevant information); one or more ICA and MIGM server computing systems 380; one or more mobile computing devices 360 (e.g., mobile image capture devices); optionally one or more camera devices 375; optionally other navigable devices 395, which receive and use floor plans and / or room / building assessment information (e.g., room and building layouts and traffic flow information) and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices); and optionally other computing systems not shown (e.g., for storing and providing additional information related to a building; for capturing building interior data; for storing and providing information to client computing devices, such as additional supplementary information associated with an image and the building or other surrounding environment it encompasses, etc.). In some embodiments, some or all of the one or more camera devices 375 can communicate directly with one or more associated mobile computing devices 360 in their vicinity (e.g., wirelessly and / or via a cable or other physical connection, and optionally in a peer-to-peer manner) (e.g., to transmit captured target images, receive instructions to initiate capture of target images or other additional data, etc.), either supplementing or replacing communication via the network 399, and wherein such associated mobile computing devices 360 are capable of providing the captured images and optionally other captured data received from one or more camera devices 375 via the network 399 to other computing systems and devices (e.g., one or more server computing systems 300 and the BUAM system 340, one or more server computing systems 380, etc.).

[0093] In the illustrated embodiment, an embodiment of the BUAM system 340 is executed in the memory 330 to perform at least some of the described techniques, such as using one or more processors 305 to execute software instructions of the system 340 in a manner that configures one or more processors 305 and one or more computing systems 300 to perform automated operations that implement those described techniques. The illustrated embodiment of the BUAM system may include one or more components (not shown) to each perform a portion of the functions of the BUAM system, and the memory may further optionally execute one or more other programs 335. As a specific example, in at least some embodiments, a copy of the ICA and / or MIGM system may be executed as one of the other programs 335, such as instead of or in addition to the ICA system 387 and the MIGM system 388 on one or more server computing systems 380. The BUAM system 340 may further store and / or retrieve various types of data on the storage device 320 (e.g., in one or more databases or other data structures) during its operation, such as information 321 about the captured room-scale images and information 323 about the captured additional images (e.g., having details about objects and / or target attributes of the objects); data 324 about the determined room layout and optionally other room-level information (e.g., traffic flow data); data 322 about additional captured data regarding the availability of objects and / or rooms and / or buildings (including ratings of target attributes of the objects); data 325 about the generated availability assessments of the objects and rooms; data 326 about the generated availability assessments of the buildings; data 328 about the intended purposes of specific types of rooms and buildings (or specific factors associated with the rooms and / or buildings); data 327 for tagging information in the images (e.g., object tag data, room tag data, etc.); and optionally various other types of additional information 329 (e.g., about users of the client computing device 390 that interacts with the BUAM system and / or operators of the mobile devices 360 and / or 375; lists or other predefined information about various types of objects expected in a certain type of room; lists or other predefined information about various types of target attributes expected in a certain type of object and optionally a certain type of room; lists or other predefined information about various types of rooms expected in a certain type of building; data about other buildings and their assessments for comparison, including ratings, etc.).The ICA system 387 and / or the MIGM system 388 can similarly store and / or retrieve various types of data on the storage device 385 (e.g., in one or more databases or other data structures) during their operation and provide some or all of this information to the BUAM system 340 for use (either in a push or pull manner), such as the image 393 (e.g., the captured 360° panoramic image), and optionally other information, such as the inter-image orientation link information 396 generated by the ICA and / or MIGM systems and used by the MIGM system to generate the floor plan; the resulting floor plan information generated by the MIGM system and optionally other building survey information 391; additional information (such as the determined room shape 392 and optionally the image position information 394) generated by the MIGM system as part of generating the floor plan; and optionally various types of additional information 397 (e.g., various analysis information related to the presentation or other use of one or more building interiors or other environments captured by the ICA system).

[0094] Some or all of the user client computing devices 390 (e.g., mobile devices), mobile computing devices 360, other navigable devices 395, and other computing systems can similarly include some or all of the same types of components shown for the server computing systems 300 and 380. As a non-limiting example, each of the mobile computing devices 360 is shown as including one or more hardware CPUs 361, I / O components 362, a storage device 365, an imaging system 364, an IMU hardware sensor 369, an optional depth sensor 363, and a memory 367 that has a BUAM application 366 and optionally one or both of a browser and one or more other client applications 368 (e.g., an application specific to the ICA system) executing within the memory 367, such as to participate in communication with the BUAM system 340, the ICA system 387, and / or other computing systems. Although specific components are not described for the other navigable devices 395 or the client computing system 390, it should be understood that they can include similar and / or additional components.

[0095] It should also be understood that the computing systems 300 and 380 and Figure 3The other systems and devices included therein are merely illustrative and are not intended to limit the scope of the present invention. The system and / or device may instead each include a plurality of interacting computing systems or devices and may be connected to other devices not specifically described, including via Bluetooth communication or other direct communication, via one or more networks (such as the Internet), via the Web, or via one or more private networks (e.g., a mobile communication network, etc.). More generally, the device or other computing system may include any combination of hardware that may optionally interact and perform functions of the described type when programmed or otherwise configured with specific software instructions and / or data structures, the hardware including but not limited to a desktop computer or other computer (e.g., a tablet computer, a slate tablet, etc.), a database server, a network storage device, and other network devices, a smartphone and other cellular phones, consumer electronic devices, wearable devices, digital music player devices, handheld gaming devices, a PDA, a wireless phone, an Internet appliance, and various other consumer products including appropriate communication capabilities. Additionally, in some embodiments, the functions provided by the illustrated BUAM system 340 may be distributed among various components, some of the described functions of the BUAM system 340 may not be provided, and / or other additional functions may be provided.

[0096] It should also be understood that although various items are described as being stored in memory or on a storage device during use, for purposes of memory management and data integrity, these items or portions thereof may be transferred between memory and other storage devices. Alternatively, in other embodiments, some or all of the software components and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Thus, in some embodiments, when configured by one or more software programs (e.g., the BUAM system 340 executed on the server computing system 300) and / or data structures, some or all of the described techniques may be performed by hardware components including one or more processors and / or memory and / or storage devices, such as by executing the stored content of software instructions including the one or more software programs and / or by storing such software instructions and / or data structures, and such as performing algorithms described in flowcharts and other disclosures herein. Additionally, in some embodiments, some or all of the systems and / or components may be implemented or provided in other ways, such as by consisting of one or more devices implemented partially or fully in firmware and / or hardware (e.g., rather than as a device implemented in whole or in part by software instructions configuring a particular CPU or other processor), including but not limited to one or more application specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), network storage device, or portable media article (e.g., DVD disk, CD disk, optical disk, flash memory device, etc.) for reading by an appropriate drive or via an appropriate connection. In some embodiments, the systems, components, and data structures may also be transmitted over a variety of computer-readable transmission media via the generated data signals (e.g., as part of a carrier wave or other analog or digital propagated signal), the transmission media including wireless and wire / cable-based media, and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may take other forms. Thus, other computer system configurations may be utilized to practice the embodiments of the present disclosure.

[0097] Figure 4 An example flowchart showing an embodiment of the ICA system routine 400 is shown. The routine may be performed by, for example Figure 1A the ICA system 160,Figure 3 performed by the ICA system 387 and / or an ICA system otherwise described herein, such as to acquire a 360° target panoramic image and / or other images within a building or other structure (e.g., for subsequent generation of a related floor plan and / or other mapping information, such as generated by embodiments of the MIGM system routines, where Figures 5A to 5C An example of such a routine is described; for subsequent evaluation of the usability of rooms and buildings, such as evaluated by embodiments of the BUAM system routines, where Figures 6A to 6B An example of such a routine is described; for subsequent determination of the acquisition location and optionally the acquisition orientation, etc. of the target image). Although examples of portions of the example routine 400 have been discussed with respect to acquiring a particular type of image at a particular location, it should be understood that this or a similar routine can be used to acquire video or other data (e.g., audio, text, etc.) and / or other types of images that are not panoramic, either in place of or in addition to such images. Additionally, although the illustrated embodiments acquire and use information from the interior of the target building, it should be understood that other embodiments can perform similar techniques for other types of data (including for non-building structures and / or for information external to one or more target buildings of interest). Additionally, some or all of the routines can be performed on a mobile device used by a user to participate in acquiring image information and / or related additional data, and / or by a system remote from such a mobile device.

[0098] The illustrated embodiment of the routine begins at block 405, where an instruction or information is received. At block 410, the routine determines whether the received instruction or information indicates collecting data representative of a building (e.g., inside a building), and if not, proceeds to block 490. Otherwise, the routine advances to block 412 to receive an indication that one or more image capture devices are ready to begin an image capture process at a first capture location, such as for a mobile computing device acting as an image capture device and / or otherwise associated with one or more separate camera devices acting as image capture devices, and wherein the image capture device is moved by a user associated therewith or by its own power or by the power of one or more other devices carrying or otherwise moving one or more image capture devices, and the received indication can be, for example, from one of the image capture devices, from another power device carrying or otherwise moving one or more image capture devices, from a user of one or more of the image capture devices, etc. After block 412, the routine advances to block 415 to perform capture location image capture activities to capture at least one 360° panoramic image for the capture location at the target building of interest by at least one image capture device (and optionally capture one or more additional images and / or other additional data by the image capture device, such as from an IMU sensor and / or depth sensor), such as to provide at least 360° of horizontal coverage about a vertical axis. The routine may also optionally obtain from the user annotations and / or other information about the capture location and / or the surrounding environment, such as for later use in presenting information about the capture location and / or the surrounding environment.

[0099] After completing block 415, the routine proceeds to block 417 to determine whether to perform a determination availability assessment at the current time based on one or more images captured in block 415, such as with respect to one or more such objects visible from the capture location and / or with respect to one or more visible target attributes of such objects and / or with respect to the room enclosing the capture location, and if so, the routine proceeds to block 419 to perform automated operations of a building availability assessment manager routine to determine availability assessment information based at least in part on the visual data of one or more images. Figures 6A to 6B An example of such a building availability assessment manager routine is shown. After block 419, the routine proceeds to block 421 to optionally display on one or more of the image capture devices (e.g., on the mobile computing device) information about the availability assessment information determined based at least in part on the visual data of one or more images, such as in some embodiments and scenarios, display one or more of the images on the mobile computing device and overlay information about the determined availability assessment information.

[0100] After block 421, or if instead it is determined in block 417 that availability assessment information is not to be determined for one or more images acquired at the current time in block 415, the routine proceeds to block 425 to determine whether there are more acquisition locations at which to acquire images, such as based on corresponding provided information (e.g., from an item in the image acquisition device, from another device that power-carries or otherwise moves one or more image devices by another device, from a user of one or more of the image acquisition devices, etc.) and / or to meet specified criteria (e.g., at least two panoramic images to be captured in each of some or all rooms in a building and / or in each of one or more areas outside the building). If so, the routine proceeds to block 427 to optionally initiate capturing link information (such as visual data, acceleration data from one or more IMU sensors, etc.) during movement of one or more image acquisition devices along a travel path away from the current acquisition location and toward the next acquisition location in the building. As described elsewhere herein, the captured link information may include additional sensor data (e.g., one or more IMUs or inertial measurement units on or otherwise carried by a user or by another power device that carries or moves one or more image acquisition devices) and / or additional visual information (e.g., panoramic images, other types of images, panoramic or non-panoramic video, etc.) recorded during such movement, and in some embodiments, the link information may be analyzed to determine the changing pose (position and orientation) of one or more image acquisition devices during movement and information about the room shape of an enclosed room (or other area) and the path of the image acquisition device during movement. Capturing of such link information may be performed in response to an explicitly received indication (e.g., from one of the image acquisition devices, from another device that power-carries or otherwise moves one or more image acquisition devices by another device, from a user of one or more of the image acquisition devices, etc.) or based on one or more automated analyses of information recorded from a mobile computing device and / or a separate camera device. Additionally, in some embodiments, the routine may also optionally determine one or more guidance cues regarding the movement of one or more image acquisition devices, the quality of sensor data, and / or the visual information captured during movement to the next acquisition location and provide them to the user (e.g., by monitoring the movement of one or more of the image acquisition devices), including information about the associated lighting / environmental conditions, the desirability of capturing the next acquisition location, and any other suitable aspects of capturing link information. Similarly, the routine may optionally (e.g., from the user) obtain annotations and / or other information regarding the travel path, such as for later use in presenting information about the travel path or the resulting panoramic-to-panoramic image connection links.In block 429, the routine then determines that one or more image capture devices have reached a next capture location (e.g., based on an indication from one of the one or more image capture devices, from another device that powers or otherwise moves one or more image capture devices by means of another device, from a user of one or more of the one or more image capture devices; based on the forward movement of the one or more image capture devices stopping within at least a predefined amount of time, etc.) to be used as a new current capture location, and returns to block 415 to perform image capture activities for the new current capture location.

[0101] If instead it is determined in block 425 that there are no longer capture locations at which to capture image information of the current building or other structure, the routine proceeds to block 430 to optionally analyze the capture location information of the building or other structure, such as identifying possible additional coverage (and / or other information) to be captured inside the building or otherwise associated with the building. For example, the ICA system may provide (e.g., to the user) one or more notifications regarding the information captured during the capture of multiple capture locations and optionally the corresponding link information, such as if it is determined that the quality of one or more segments of the recorded information is insufficient or not as desired or does not appear to provide complete building coverage, or alternatively may provide (e.g., to the user, to the device that powers or otherwise moves one or more image capture devices, etc.) corresponding recapture instructions. Additionally, in at least some embodiments, if the captured images (e.g., at least two panoramic images in each room, panoramic images within a maximum specified distance of each other, etc.) do not meet the minimum criteria of the images (e.g., minimum number and / or type of images), the ICA system may prompt or indicate to capture additional panoramic images in a similar manner to meet such criteria. After block 430, the routine continues to block 435 to optionally preprocess the captured 360° target panoramic images, which are then subsequently used to generate relevant mapping information (e.g., to put the images in a spherical format, determine the vanishing lines and vanishing points of the images, etc.). In block 480, these images and any associated generated or obtained information are stored for later use.

[0102] If it is determined in block 410 that the instruction or other information recited in block 405 is not to collect images and other data representing a building, the routine proceeds to block 490 to perform any other indicated operations as appropriate (such as, any household task), configure parameters for use in the various operations of the system (e.g., at least in part based on information specified by a user of the system, such as a user of an image capture device that captures images inside one or more buildings, an operator user of the ICA system, etc.), obtain and store additional information about the user of the system, and respond to requests for the generated and stored information (e.g., requests for use of such information by the MIGM system and / or the BUAM system, requests for use of such information by a building map viewer system or for display or other presentation by other systems, requests from one or more client devices for display or other presentation of such information, operations for generating and / or training one or more neural networks or another analysis mechanism for automated operation of the routine, etc.).

[0103] After block 480 or 490, the routine proceeds to block 495 to determine whether to continue, such as until a clear termination indication is received, or alternatively only if a clear continue indication is received. If it is determined to continue, the routine returns to block 405 to wait for additional instructions or information, and if not, proceeds to step 499 and ends.

[0104] Figures 5A to 5C An example embodiment of a flowchart for a Mapping Information Generation Manager (MIGM) system routine 500 is shown. The routine can be performed by, for example Figure 1A the MIGM system 160 Figure 3 the MIGM system 388 and / or the MIGM system as described elsewhere herein, such as by analyzing information from one or more images collected in a room and combining it to determine the shape of the room (or other defined area), generating a floor plan of a building or other defined area at least in part based on one or more images of the area and optionally additional data captured by a mobile computing device, and / or generating other mapping information of a building or other defined area at least in part based on one or more images of the area and optionally additional data captured by a mobile computing device. In Figures 5A to 5CIn the example, the determined room shape of the room can be a 3D fully enclosed combination of planar surfaces representing the walls, ceiling, and floor of the room, or can have other forms (e.g., at least partially based on a 3D point cloud), and the generated survey information of the building (e.g., a house) includes a 2D floor plan and / or a 3D computer model floor plan and / or a set of linked images, where the links indicate the relative orientation between various pairs of images. However, in other embodiments, other types of room shapes and / or survey information can be generated and used in other ways, including other types of structures and defined areas, as discussed elsewhere herein.

[0105] The illustrated embodiment of the routine begins at block 505, where information or instructions are received. The routine continues to block 510 to determine whether image information is currently available for analysis for one or more rooms (e.g., for some or all of the indicated building), or instead to acquire such image information currently. If it is determined in block 510 that some or all of the image information is to be acquired currently, the routine continues to block 512 to acquire such information, optionally waiting for one or more image acquisition devices to move through one or more rooms of the building (e.g., carried by an associated user or powered by one or more other devices that move the one or more image acquisition devices by themselves or carried or otherwise moved) and acquiring panoramic or other images at one or more acquisition locations in one or more of the rooms (e.g., multiple acquisition locations in each room of the building), optionally together with metadata information regarding acquisition and / or interconnection information related to the movement between the acquisition locations, as discussed in more detail elsewhere herein. Figure 4 An example embodiment of an ICA system routine for performing such image acquisition is provided. If instead it is determined in block 510 that images are not being acquired currently, the routine instead continues to block 515 to obtain existing panoramic or other images from one or more acquisition locations in one or more of the rooms (e.g., multiple acquisition locations in each room of the building), optionally together with metadata information regarding acquisition and / or interconnection information related to the movement between the acquisition locations, such as may have been supplied in block 505 together with corresponding instructions and / or previously obtained by the ICA system in some cases.

[0106] After block 512 or 515, the routine continues to block 520 where it is determined whether to generate a set of linked target panoramic images (or other images) of a building or other group of rooms, and if so, continues to block 525. In block 525, the routine selects pairs of at least some of the images (e.g., based on pairs of images having overlapping visual content and / or based on link information connecting pairs of images), and for each pair, determines the relative orientation between the pair of images based on the shared visual content and / or other captured link interconnect information (e.g., movement information) related to the pair of images (whether the movement is directly from the acquisition location of one image of the pair to the acquisition location of the other image of the pair, or instead via one or more other intermediate acquisition locations of other images between those start and end acquisition locations). In block 525, the routine further determines the global relative positions of some or all of the images with respect to each other in a common coordinate system using at least the relative orientation information of the image pairs, such as to create a virtual tour through which an end user can move from any one of the images to one or more other images linked to the starting image (e.g., via user-selectable controls that select an image display (such as an overlay on the displayed image) for each such other linked image), and similarly from the next image to one or more additional images linked to that next image, etc. Additional details regarding creating such a set of linked images are included elsewhere in this document.

[0107] After block 525, or if instead in block 520 it is determined that the instruction or other information received in block 505 is not to determine a set of linked images, the routine continues to block 530 to determine whether the instruction received in block 505 indicates determining the shape of one or more rooms from previously or currently acquired images in the room (e.g., from one or more panoramic images acquired in each of the rooms), and if so, continues to block 550, otherwise continues to block 590.

[0108] In block 550, the routine advances to selecting the next room (starting with the first) in which one or more panoramic images and / or other images can be obtained that were captured in the room, and determining the 2D and / or 3D shape of the room based at least in part on visual data of one or more images acquired in the room and / or additional data acquired in the room, including optionally obtaining additional metadata for each image (e.g., acquisition height information of the camera device or other image acquisition device used to acquire the image). Determining the room shape of the room can include analyzing the visual content of one or more images acquired in the room by one or more image acquisition devices and / or analyzing additional non-visual data acquired in the room (e.g., by one or more image acquisition devices), including determining initial estimated acquisition pose information (e.g., acquisition location and optionally acquisition orientation) for each of the images. Analysis of the various data acquired in the room can also include identifying wall structure element features of the room (e.g., windows, doorways, and stairways, as well as other inter-room wall openings and connecting passageways, wall boundaries between a wall and another wall and / or receptacle and / or floor, etc.) and determining the locations of those identified features within the determined room shape of the room, optionally by generating a 3D point cloud of some or all of the room walls and optionally the ceiling and / or floor (e.g., by at least analyzing the visual data of the images acquired in the room and optionally additional data captured by one or more of the image acquisition devices, such as using one or more of SfM or SLAM or MVS analysis), and / or by determining planar surfaces corresponding to some or all of the walls and optionally the floor and / or ceiling (e.g., by determining the normal / orthogonal direction of the plane of the identified features and combining such information to determine wall position hypotheses and optionally clustering multiple wall position hypotheses for a given wall to achieve a final determination of the position of the wall). Additional details regarding determining room shape and identifying additional information about the room, including initial estimated acquisition pose information of images acquired in the room, are included elsewhere in this document.

[0109] After block 550, the routine proceeds to block 567 to determine whether there are more rooms for which one or more captured images are available, and if so, returns to block 550 to determine the room shape of the next such room. Otherwise, the routine proceeds to block 537 to determine whether to generate a floor plan of the indicated building (e.g., at least in part based on the room shapes determined in block 550), and if not, proceeds to block 590. Otherwise, the routine proceeds to block 537, where the routine optionally obtains additional information about the building, such as from activities performed during the acquisition and optionally analysis of the images, and / or from one or more external sources (e.g., online databases, information provided by one or more end users, etc.), such additional information can include, for example, the external dimensions and / or shape of the building, additional images and / or annotation information acquired corresponding to specific locations within the building (optionally for locations different from the acquisition location of the captured panoramic or other images), additional images and / or annotation information acquired corresponding to specific locations outside the building (e.g., around the building and / or for other structures on the same property), etc.

[0110] After block 537, the routine proceeds to block 575 to retrieve the room shape (e.g., the room shape generated in block 550) or otherwise obtain the room shape of the rooms of the building (e.g., based on human-supplied input), whether it is a 2D or 3D room shape. The routine then proceeds to block 577, where the routine uses the determined room shape to create an initial 2D floor plan, such as by connecting the room-to-room passages in their respective rooms, by optionally positioning the room shape around the determined acquisition location of the image (e.g., if the acquisition locations are interconnected), and by optionally applying one or more constraints or optimizations. Such a floor plan may include, for example, the relative positions and shape information of various rooms, without providing any actual dimension information of the individual rooms or the building as a whole, and may further include multiple linked or associated sub-maps of the building (e.g., to reflect different floors, levels, sections, etc.). The routine further associates the positions of doors, wall openings, and other identified wall elements on the floor plan. After block 577, the routine optionally performs one or more of steps 580 to 583 to determine additional information and associate it with the floor plan. In block 580, the routine optionally estimates the dimensions of some or all of the rooms, such as based on the analysis of the image and / or its acquisition metadata or based on the overall dimension information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan. After block 580, the routine proceeds to block 583 to optionally associate further information with the floor plan (e.g., having a specific room or other location within the building), such further information being additional images and / or annotation information for the specified location. In block 585, if the room shape from block 575 is not a 3D room shape, the routine further estimates the height of the walls in some or all of the rooms, such as based on the analysis of the image and optionally the size of known objects in the image and the height information of the camera at the time of image acquisition, and uses this height information to generate a 3D room shape of the rooms. The routine further uses the 3D room shape (whether from block 575 or block 585) to generate a 3D computer model floor plan of the building, where the 2D and 3D floor plans are associated with each other. In block 485, the routine then optionally calls the MIGM system to perform further analysis using the information obtained and / or generated in routine 400, such as to generate a partial or complete floor plan of the building and / or generate other surveying-related information, and where Figures 5A to 5C routine 500 provides an example of such a routine of the MIGM system. It should be understood that if sufficiently detailed dimension information is available, architectural drawings, blueprints, etc. can be generated from the floor plan.

[0111] After block 585, the routine continues to block 588 to store the determined one or more room shapes and / or the generated survey information and / or other generated information, and optionally further use some or all of the determined and generated information, such as to provide the generated 2D floor plan and / or 3D computer model floor plan for display on one or more client devices and / or provide to one or more other devices for automated navigation of those devices and / or associated vehicles or other entities, to similarly provide and use information about the determined room shapes and / or a set of linked panoramic images and / or additional information determined based on the contents of the rooms and / or the passageways between the rooms, to provide the information as a response to another routine that called routine 500, etc.

[0112] If it is determined in block 530 that the information or instruction received in block 505 is not for determining the shape of one or more rooms, or if it is determined in block 535 that the information or instruction received in block 505 is not for generating a floor plan of the indicated building, the routine continues to block 590 to perform one or more other indicated operations as appropriate. Such other operations can include, for example, receiving and responding to requests for previously generated floor plans and / or previously determined room shapes and / or previously determined sets of linked images and / or other generated information (e.g., requests for such information for automated navigation by one or more other devices, requests for such information for use by the BUAM system, requests for such information for display or other rendering by a building map viewer system or other system, requests for display or other rendering of such information from one or more client devices, operations for generating and / or training one or more neural networks or another analysis mechanism for automated operation of the routine, etc.), obtaining and storing information about the building for later operations (e.g., information about the dimensions, number, or type of rooms, total floor area, other adjacent or nearby buildings, adjacent or nearby vegetation, external images, etc.).

[0113] After block 588 or 590, the routine continues to block 595 to determine whether to continue, such as until a clear termination indication is received, or alternatively to continue only if a clear continue indication is received. If it is determined to continue, the routine returns to block 505 to wait and receive additional instructions or information, otherwise, it continues to block 599 and ends.

[0114] Although not with respect to in Figures 5A to 5Cillustrated in the example embodiments, but in some embodiments, a human user may further assist in facilitating some operations of the MIGM system, such as an operator user and / or an end user of the MIGM system providing one or more types of input for subsequent automated operations. As a non-exclusive example, such a human user may provide one or more types of input as follows: providing input to assist in linking a set of images, such as providing input in block 525 that is used as part of the automated operation of that block (e.g., to specify or adjust an initially automatically determined orientation between one or more pairs of images, to specify or adjust an initially automatically determined final global position of some or all of the images relative to each other, etc.); providing input in block 537 that is used as part of a subsequent automated operation, such as one or more of the types of information shown about the building; providing input relative to block 550 that is used as part of a subsequent automated operation, such as to specify or adjust an automatically determined pose information of one or more of the images, to specify or adjust an initially automatically determined information about acquisition height information of one or more of the images and / or other metadata, to specify or adjust an initially automatically determined element position and / or an estimated room shape, etc.; providing input relative to block 577 that is used as part of a subsequent operation, such as to specify or adjust an initially automatically determined position of a room shape within the generated floor plan and / or to specify or adjust the initially automatically determined room shape itself within such a floor plan; providing input relative to one or more of blocks 580 and 583 and 585 that is used as part of a subsequent operation, such as to specify or adjust an initially automatically determined information of one or more types discussed relative to those blocks, etc. Additional details regarding embodiments in which one or more human users provide input that is further used for additional automated operations of the MIGM system are included elsewhere in this document.

[0115] Figures 6A to 6B illustrates an example flowchart for a Building Usability Assessment Manager (BUAM) system routine 600. The routine may be executed, for example, Figure 1A by the BUAM system 140 and / or the BUAM application 156, Figure 3 the BUAM system 340 and / or the BUAM application 366, and / or as executed by a BUAM system as described Figures 2P to 2X herein and elsewhere in this document, such as to perform automated operations related to analyzing visual data from images captured in a room of a building and optionally additional captured data to evaluate the rooms of the building and the objects in the rooms (e.g., at least in part based on an automated assessment of one or more target attributes of each object) and the room layout and other usability information of the building itself. In Figures 6A to 6BIn the example, the captured data of the specified type is used and the analysis is performed in a specific manner. However, in other embodiments, other types of information can be obtained, analyzed, and otherwise used in other ways. Additionally, although the illustrated embodiments collect and use information from the interior of the target building, it should be understood that other embodiments can perform similar techniques for other types of data (including information for non-building structures and / or outside of one or more target buildings of interest). Additionally, some or all of the routines can be executed on a mobile device used by the user (e.g., a mobile computing device or other image acquisition device) to participate in collecting image information and / or related additional data, and / or be executed by a system remote from such a mobile device.

[0116] The illustrated embodiment of the routine begins at block 605, where information or an instruction is received. The routine proceeds to block 610 to determine whether the instruction or other information indicates an assessment of the availability of one or more indicated rooms (e.g., for some or all of the rooms of a building). If not, the routine proceeds to block 690, and otherwise proceeds to block 615, where the routine selects the next indicated room (starting with the first), and obtains one or more initial images of the room whose visual data includes most or all of the room (e.g., images previously acquired by an ICA system; images acquired simultaneously in the room by one or more image acquisition devices, such as in an automated manner and / or using one or more associated users participating in the image acquisition, and optionally in response to corresponding instructions initiated by the routine and provided to the image acquisition device and / or the associated users, etc.). One or more initial images are then analyzed to identify one or more objects of interest in the room for which more data is to be collected (if the one or more initial images do not include sufficient detail about the object in their visual data, if no other types of data other than visual data are available in the information obtained about the one or more images and to be captured, etc.), and optionally to determine additional information about the room, such as to evaluate a label or other type or category information of the room, to evaluate the shape of the room and / or the layout of the items in the room, to evaluate the expected traffic flow pattern of the room (e.g., at least in part based on the layout and / or shape) and / or the actual communication traffic pattern of the room (e.g., if there are sufficient images to show people moving through the room), etc. In some embodiments, additional information about some or all of the objects, such as object location, object label or other type or category information of the object, is additionally determined while identifying the objects (e.g., in a joint manner or other relevant manner) at least in part based on the analysis of the visual data. Alternatively, in some embodiments and situations, at least some of this information (e.g., one or more labels or other type or category information) may be obtained in other ways for the room and / or one or more objects in the room, such as from previously generated information (e.g., generated by an ICA system) and / or from simultaneously generated information (e.g., at least in part based on information from one or more users participating in simultaneous image acquisition in the room, etc.). Additional details regarding determining information about a room based at least in part on visual data of one or more initial images captured in the room are included elsewhere herein.

[0117] After block 615, the routine proceeds to block 620 where the routine selects the next object (starting with the first) identified in block 615 of the current room and determines additional information about the object, such as determining the type and / or category of the object (if not already determined in block 615) by analyzing visual data in one or more initial images and optionally using other data, to determine one or more target properties of interest for the object for which additional data is to be collected (e.g., at least in part based on the type or category of the object, such as a predefined list of some or all such target properties for objects of that type or category), to determine the location of the object (if not already determined in block 615), etc. As part of doing so, the routine may further analyze the visual data of one or more initial images to verify whether the visual data includes sufficient detail about each of the target properties, and if sufficient detail is already available, exclude the target properties from the additional data to be captured (and other non-visual types of additional data are not identified as to be captured). The routine further determines one or more types of additional data to be collected for each of the target properties that are not yet available (whether one or more additional images or one or more other types of additional data), generates corresponding instructions to indicate automated capture of the additional data and / or indicate associated user participation in the capture of the additional data, and provides the instructions to one or more image capture devices and / or users, optionally along with examples (or access to such examples if needed, such as via a user-selectable link). The routine then further acquires the additional data about the object and its target properties from one or more image capture devices and / or associated users. Additional details about determining information about an object based at least in part on visual data of one or more initial images captured for the room in which the object is located are included elsewhere in this document.

[0118] After block 620, the routine proceeds to block 625 where the routine optionally evaluates the additional data captured in block 620 to identify possible problems (e.g., incorrect objects and / or target attributes visible in the additional image and / or described in other data, insufficient visual data in the additional image or other data to enable assessment of the target attributes and / or evaluation of the objects, other types of image problems or other types of data problems, etc.), and if so, a corrective action may be initiated (e.g., providing additional instructions to one or more image acquisition devices and / or associated users to capture additional images and / or other data to correct the problem), including obtaining any such corrective additional images and / or other data used to supplement or replace the initial additional image and / or other initial additional data with the identified possible problem. Additionally, although the acquisition of the initial image, additional images, and optionally other additional data is illustrated in blocks 615 to 625 as occurring after providing the instructions and before proceeding to the next block of the routine, it should be understood that the obtaining of such images and / or other data may occur substantially immediately (e.g., concurrently with the instructions, such as interactively) and / or asynchronously (e.g., a significant amount of time after providing the instructions, such as minutes, hours, days, etc.), and the routine may perform other operations (e.g., for other rooms and / or other buildings) while waiting for the images and optionally other additional data.

[0119] After block 625, the routine proceeds to block 630 where the routine determines whether there is an identified object in the current room and if so, returns to block 620 to select the next such object. Otherwise, the routine proceeds to block 635 where the routine analyzes the captured additional data available for each identified target attribute of each of the identified objects in the current room in order to evaluate the target attribute of the object against one or more defined factors of the type of target attribute and / or object or other defined attribute criteria, and in at least some embodiments, performs an assessment of the target attribute to estimate the current contribution of the target attribute to the usability assessment of the object, such as the contribution of the object to the overall usability of the room for the intended purpose of the room. After block 635, the routine proceeds to block 640 where, for each identified object in the current room, the captured additional data and other information about the room are analyzed to evaluate the usability of the object against one or more defined object criteria of the type of object and / or the enclosed room, such as to evaluate the contribution of the object to the overall usability of the room for the intended purpose of the room, including using information from the assessment of one or more target attributes of the object (e.g., combining assessment information from multiple target attributes) and optionally further using additional information about the object. In block 645, the routine then evaluates the overall usability of the room for the intended purpose, such as evaluating against one or more defined room criteria of the type of room, including using information from the assessment of one or more identified objects in the room (e.g., combining assessment information from multiple identified objects), and optionally further using additional information about the room (e.g., the evaluated room layout, the estimated traffic flow pattern of the room, etc.).

[0120] After block 645, the routine proceeds to block 650 where the routine determines whether there is an additional room to be evaluated and, if so, returns to block 615 to select the next such room. In at least some embodiments and scenarios, the determination of whether there is an additional room can be made at least in part dynamically based on one or more image capture devices and / or associated users in the room, such as if as part of the next iteration of the operation of block 615, one or more image capture devices and / or associated users move to the next room in the building and interactively proceed to obtain one or more initial images of the next room (or alternatively indicate that the last room of the building has been captured such that there are no more rooms). Otherwise, the routine proceeds to block 685 where, if a plurality of rooms of the building have been evaluated, the routine optionally evaluates the overall usability of the building relative to one or more defined building evaluation criteria (such as relative to the intended purpose of the building). As part of doing so, the routine can use information from the evaluations of the rooms in the building (e.g., to combine evaluation information from multiple rooms), and further optionally use additional information about the building (e.g., an evaluated building layout, an estimated traffic flow pattern of the building, etc.). Additional details regarding the assessments and evaluations performed relative to blocks 635 - 645 and 685 are included elsewhere herein.

[0121] After block 685, the routine proceeds to block 688 where the routine stores the information determined and generated in blocks 615 - 685 and optionally displays some or all of the determined and / or evaluated information and / or optionally provides some or all of the determined and / or evaluated information for further use (e.g., for automated operation of the building by one or more devices; in response to another routine that calls routine 600, such as relative to Figure 4 block 419, etc.).

[0122] If it is determined in block 610 that the instruction or information received in block 605 is not to evaluate the availability of one or more indicated rooms, the routine proceeds to block 690, where the routine performs one or more other indicated operations as appropriate. Such other operations can include, for example, one or more of the following: receiving and responding to requests for evaluations and / or other generated information for previously generated rooms, buildings, and / or objects (e.g., requests for such information for use by one or more other devices for automated navigation, requests for such information for display or other rendering by a building map viewer system or other system, requests from one or more client devices for display or other rendering of such information, etc.); operations for generating and / or training one or more neural networks or other analysis mechanisms for automated operation of the routine; obtaining and storing information about buildings, rooms, objects, and / or object target attributes for later operations (e.g., information about expected or typical target attributes of a particular object or object type, information about expected or typical objects in a particular room or room type, information about expected or typical rooms in a particular building or building type, information about one or more types of defining criteria for automated analysis, information about factors for rating a particular target attribute or target attribute type or the type of associated object, etc.); information about the expected purposes of particular rooms and / or room types and / or buildings and / or building types, etc.

[0123] After block 688 or 690, the routine proceeds to block 695, where the routine determines whether to continue, such as until a clear termination indication is received, or alternatively not to continue unless a clear continue indication is received. If it is determined to continue, the routine returns to block 605; otherwise, it proceeds to block 699 and ends. It should be understood that although the example embodiment of routine 600 receives information about evaluating one or more rooms and optionally a multi-room building and proceeds to perform such activities, other embodiments of the routine can analyze other levels of information, such as alternatively rating one or more indicated target attributes (e.g., without further evaluating one or more objects corresponding to the target attribute), evaluating one or more indicated objects (e.g., without further evaluating the one or more rooms in which the one or more objects are located), etc.

[0124] Figure 7 An example embodiment of a flowchart for a routine 700 for a building map viewer system is shown. The routine can be implemented, for example, by a Figure 1A map viewer client computing device 175 and its one or more software systems (not shown), Figure 3performed by the client computing device 390 and / or the mobile computing device 360, and / or a mapping information viewer or rendering system as described elsewhere herein, such as: receiving and displaying the determined room shape and / or other mapping information (e.g., 2D or 3D floor plan) of a defined area, the mapping information optionally including one or more determined image capture locations and / or one or more visual indications of the generated availability assessments (e.g., associated with a particular location in the mapping information); and optionally displaying additional information associated with a particular location (e.g., an image, optionally together with availability assessment information superimposed on and / or otherwise associated with one or more objects and / or the room visible in the image) in the mapping information. In Figure 7 an example, the presented mapping information is for the interior of a building (such as a house), but in other embodiments, other types of mapping information may be presented and otherwise used for other types of buildings or environments, as discussed elsewhere herein.

[0125] The illustrated embodiment of the routine begins at block 705, where an instruction or information is received. At block 710, the routine determines whether the received instruction or information indicates to display or otherwise present information representing the interior of a building, and if not, proceeds to block 790. Otherwise, the routine advances to block 712 to retrieve one or more room shapes or floor plans of the building or other generated survey information of the building, and optionally an indication of associated link information of the surrounding location inside and / or outside the building, and optionally an indication of information (e.g., usability assessment information) to be superimposed on the survey information or otherwise associated with the survey information, and selects an initial view of the retrieved information (e.g., a view of the floor plan, a particular room shape, etc.). In block 715, the routine then displays or otherwise presents the current view of the retrieved information and waits for a user selection in block 717. After the user selection in block 717, if it is determined in block 720 that the user selection corresponds to adjusting the current view of the current location (e.g., changing one or more aspects of the current view, adding usability assessment information superimposed on or otherwise associated with the current view, etc.), the routine proceeds to block 722 to update the current view according to the user selection and then returns to block 715 to update the information displayed or otherwise presented accordingly. The user selection and corresponding update of the current view may include, for example, displaying or otherwise presenting an associated link information of the user selection (e.g., a particular image associated with the displayed visual indication of the determined acquisition location, such as superimposing the associated link information on at least some of the previously displayed ones) and / or changing the way the current view is displayed (e.g., zooming in or out; rotating the information when appropriate; selecting a new portion of the floor plan to be displayed or otherwise presented, such as where some or all of the new portion was previously invisible, or instead where the new portion is a subset of the previously visible information, etc.).

[0126] If instead it is determined in block 710 that the instruction or other information received in block 705 will not present information representing the interior of a building, the routine instead proceeds to block 790 to perform other indicated operations as appropriate (such as, any household task), configure parameters for use in the various operations of the system (e.g., at least in part based on information specified by a user of the system, the user such as a mobile device user capturing one or more building interiors, an operator user of the MIGM system, etc., including personalizing the display of information according to the preferences of a particular user), obtain and store other information about the system user, respond to requests for the generated and stored information, etc.

[0127] After box 790, or if it is determined in box 720 that the user selection does not correspond to the current building area, the routine proceeds to box 795 to determine whether to continue, such as until a clear termination indication is received, or alternatively only if a clear continue indication is received. If it is determined to continue (including if the user makes a selection in box 717 related to the new location to be presented), the routine returns to box 705 to wait for additional instructions or information (or proceeds directly to box 712 if the user makes a selection in box 717 related to the new location to be presented), and if not, proceeds to step 799 and ends.

[0128] The non-exclusive example embodiments described herein are further described in the following clauses.

[0129] A01. A computer-implemented method for performing automated operations by one or more computing systems, the method comprising:

[0130] Obtaining, by one or more computing systems, one or more images captured in a room of a house and information about a plurality of objects installed in the room;

[0131] Determining, by the one or more computing systems and for each of the plurality of objects, one or more target attributes of the object for which additional data is to be captured, including analyzing visual data of the one or more images to identify that the visual data lacks details sufficient to meet a threshold defining the one or more target attributes of the object;

[0132] Providing, by the one or more computing systems, instructions for capturing the additional data for the one or more target attributes of each of the plurality of objects, wherein capturing the additional data includes obtaining additional visual data in one or more additional images of the plurality of objects of one or more specified types;

[0133] Analyzing, by the one or more computing systems and via use of at least one trained neural network, the additional visual data of the additional images to verify, for each of the plurality of objects, that the additional data for the one or more target attributes of the object has been captured;

[0134] Determining, by the one or more computing systems, an assessment of the current availability of the room for an indication purpose, including assessing the contribution of the plurality of objects to the current availability of the room based at least in part on information from the additional data for the one or more target attributes of each of the plurality of objects, and including combining information about the assessed contribution of the plurality of objects; and

[0135] The one or more computing systems provide information evaluating the determination of the current availability of the room.

[0136] A02. A computer-implemented method for one or more computing systems to perform automated operations, the method comprising:

[0137] Obtaining, by the one or more computing systems, one or more panoramic images captured in a room of a house and having visual data, the one or more panoramic images combined to provide 360-degree horizontal visual data of the room;

[0138] Determining, by the one or more computing systems and for each of a plurality of objects installed in the room, one or more target attributes of the object for which additional data is to be captured, including analyzing the visual data of the one or more panoramic images to identify the plurality of objects and determine the locations of the plurality of objects in the room, and determining that the visual data of the one or more panoramic images lacks details sufficient to meet a threshold defining the one or more target attributes for each of the plurality of objects;

[0139] Providing, by the one or more computing systems, instructions for capturing the additional data regarding the one or more target attributes for each of the plurality of objects and the determined locations of the plurality of objects, wherein capturing the additional data includes obtaining one or more additional stereoscopic images of the specified type of the plurality of objects;

[0140] Analyzing, by the one or more computing systems and via using at least one trained neural network, the visual data of the additional stereoscopic images to verify, for each of the plurality of objects, whether the additional data regarding the one or more target attributes of the object has been captured, including determining, for one of the additional stereoscopic images, that the one additional stereoscopic image lacks any of the additional data due to failure to include visual data sufficient to meet the threshold defining any of the target attributes for any of the plurality of objects;

[0141] Providing, by the one or more computing systems, other instructions to recapture the one additional stereoscopic image and obtaining a new copy of the one additional stereoscopic image including at least some of the additional data;

[0142] An evaluation by the one or more computing systems of the current availability of the room for the indicated purpose, including evaluating the current contribution of the plurality of objects to the current availability of the room at least in part based on information from the additional data regarding one or more target attributes of each of the plurality of objects, and including combining information regarding the evaluated contributions of the plurality of objects; and

[0143] The one or more computing systems display information regarding the determined evaluation of the current availability of the room and additional visual information regarding the room.

[0144] A03. A computer-implemented method for one or more computing systems to perform automated operations, the method comprising:

[0145] The one or more computing systems obtain one or more images captured in a room of a building;

[0146] The one or more computing systems determine a plurality of objects in the room and at least in part based on analyzing visual data of the one or more images;

[0147] The one or more computing systems provide instructions for capturing additional data regarding one or more target attributes of each of the plurality of objects, including obtaining additional visual data in additional images of the plurality of objects;

[0148] The one or more computing systems analyze the additional visual data of the additional images to verify for each of the plurality of objects whether the additional data regarding the one or more target attributes of the object has been captured, including determining that the additional visual data lacks visual data that meets a defined threshold for at least one target attribute of one of the plurality of objects;

[0149] The one or more computing systems provide other instructions to capture yet another additional image having additional visual data that meets the defined threshold for the at least one target attribute of the one object, and obtain the yet another additional image;

[0150] The one or more computing systems determine, for each of the plurality of objects, an evaluation of the contribution of the object to the availability of the room for the indicated purpose at least in part based on information from the additional data regarding the one or more target attributes of the object; and

[0151] The one or more computing systems provide information regarding the determined evaluation of each of the plurality of objects.

[0152] A04. A computer-implemented method for performing automated operations by one or more computing systems, the method comprising:

[0153] Obtaining one or more images captured in a room of a building;

[0154] Determining one or more objects in the room at least in part based on analyzing visual data of the one or more images, and determining one or more target attributes for each of the one or more objects for which additional data is to be captured at least in part based on the visual data of the one or more images lacking details sufficient to meet a threshold defining those target attributes;

[0155] Providing instructions for capturing the additional data for each of the one or more objects, including capturing additional visual data about the one or more objects;

[0156] Analyzing the additional visual data for each of the one or more objects as part of verifying that the additional data for the one or more target attributes of the object has been captured; and

[0157] Providing information about the verification for each of the one or more objects.

[0158] A05. The computer-implemented method according to any one of clauses A01 to A04, wherein providing the instructions for capturing the additional data further comprises providing instructions to obtain at least one of the type of the room or a label of the room, and for each of the plurality of objects, each of the type of the object or a label of the object, and for one or more of the plurality of objects, a text description of one or more additional attributes of the one or more objects, and wherein the assessment of the current availability of the room and the assessment of the current contribution of the plurality of objects are determined at least in part based on the obtained additional information.

[0159] A06. The computer-implemented method according to any one of clauses A01 to A05, wherein the house comprises a plurality of rooms and has one or more associated external areas outside the house, and wherein the method further comprises:

[0160] Performing, by the one or more computing systems and for each of the plurality of rooms, obtaining and determining the one or more target attributes of each of the plurality of target objects, and providing the instructions, and analyzing, and providing the other instructions, and determining the assessment of the current availability of the room;

[0161] Performing, by the one or more computing systems and for each of the one or more associated external areas, obtaining one or more other images captured in the external area, and determining one or more other target attributes of each of the one or more other objects in the external area, and providing additional instructions for providing additional visual data in additional images capturing the one or more other target attributes of each of the one or more other objects, and evaluating the availability of the external area for another indication purpose at least in part based on combining the other determined contributions of the one or more other objects; and

[0162] Determining, by the one or more computing systems, an evaluation of the overall availability of the house at least in part based on combining the information of the determined evaluation of the availability for each of the plurality of rooms with the information of the determined evaluation of the current availability for each of the one or more external areas,

[0163] And wherein displaying the information by the one or more computing systems further includes displaying the information of the determined evaluation of the overall availability of the house superimposed on a displayed floor plan of the house.

[0164] A07. The computer-implemented method according to any one of clauses A01 to A06, further comprising analyzing, by the one or more computing systems, the visual data of the one or more images to determine the type of the object for each of the plurality of objects, and wherein the one or more target attributes of each of the plurality of objects for which additional data is to be captured are determined at least in part based on the determined type of the object.

[0165] A08. The computer-implemented method according to any one of clauses A01 to A07, further comprising:

[0166] Obtaining, by the one or more computing systems, information about labels identifying the plurality of objects; and

[0167] Identifying, by the one or more computing systems and based on the labels, a plurality of target attributes of the plurality of objects, the plurality of target attributes including the one or more target attributes of each of the plurality of objects.

[0168] A09. The computer-implemented method according to any one of clauses A01 to A08, further comprising:

[0169] Obtaining, by the one or more computing systems, information about at least one of a label of the room or a type of the room; and

[0170] Determine the plurality of objects by the one or more computing systems and at least in part based on at least one of the label of the room or the type of the room.

[0171] A10. The computer-implemented method according to any one of clauses A01 to A09, wherein the additional data includes information about at least one of the shape of the room or the layout of the items in the room, and wherein the evaluation of the current availability of the room for the indicated purpose is determined at least in part based on the information about at least one of the shape or layout of the room.

[0172] A11. The computer-implemented method according to any one of clauses A01 to A10, wherein the additional data includes at least one of the type of the room or the label of the room, and wherein the method further includes determining, by the one or more computing systems, the purpose of the indication of the room at least in part based on at least one of the type of the room or the label of the room.

[0173] A12. The computer-implemented method according to any one of clauses A01 to A11, wherein analyzing the additional visual data of the additional image further includes determining that the additional visual data lacks visual data that meets a defined threshold of at least one target attribute of one of the plurality of objects, and wherein the method further includes providing, by the one or more computing systems, other instructions to capture another additional image having additional visual data that meets the defined threshold of at least one target attribute of the one object, and using the other additional visual data as part of evaluating the contribution of the one object to the current availability of the room.

[0174] A13. The computer-implemented method according to any one of clauses A01 to A12, wherein analyzing the additional visual data of the additional image further includes determining that one of the additional images lacks any of the additional data, and wherein the method further includes providing, by the one or more computing systems, other instructions to capture another copy of the one additional image including at least some of the additional data, and using the other copy of the additional image as part of evaluating the contribution of the plurality of objects to the current availability of the room.

[0175] A14. The computer-implemented method according to any one of clauses A01 to A13, wherein analyzing the visual data of the one or more images further includes determining the positions of the plurality of objects in the one or more images, and wherein providing the instructions further includes providing information about the determined positions of the plurality of objects.

[0176] A15. The computer-implemented method according to any one of clauses A01 to A14, wherein each of the plurality of objects installed in the room is at least one of the following: a lighting fixture, or a sanitary fixture, or built-in furniture, or a built-in structure inside the wall of the room, or an electrical appliance, or a gas-powered appliance, or an oil-powered appliance, or an object that generates fire by burning fuel, or an installed floor, or an installed wall covering, or an installed window covering, or the hardware attached to a door, or the hardware attached to a window, or an installed countertop.

[0177] A16. The computer-implemented method according to any one of clauses A01 to A15, wherein the visual data for analyzing the one or more images further identifies one or more additional objects in the room that are each a piece of furniture or a movable item, and wherein for each of the one or more additional objects, determining the one or more target attributes, providing the instructions, analyzing the additional visual data, and evaluating the contribution are further performed, and wherein the evaluation of the current availability of the room for the indicated purpose is further determined at least in part based on the evaluation of the contribution of the one or more additional objects to the current availability of the room.

[0178] A17. The computer-implemented method according to any one of clauses A01 to A16, wherein determining the evaluation of the availability of the room for the indicated purpose at least in part based on information combining the contributions of the evaluation of the plurality of objects includes at least one of the following: performing a weighted average of the contributions of the evaluation of the plurality of objects by the one or more computing systems using weights at least partially based on the types of the plurality of objects, or providing the contributions of the evaluation of the plurality of objects to an additional trained neural network, and receiving the determined evaluation of the current availability of the room from the additional trained neural network.

[0179] A18. The computer-implemented method according to any one of clauses A01 to A17, wherein evaluating the contribution of the plurality of objects to the current availability of the room for the indicated purpose and determining the evaluation of the current availability of the room for the indicated purpose are performed by evaluating the plurality of objects and the characteristics of the room including at least one of condition, quality, functionality, or effectiveness.

[0180] A19. A computer-implemented method according to any one of clauses A01 to A18, wherein the one or more images include one or more panoramic images, and the one or more panoramic images combined include a 360-degree horizontal visual coverage of the room, wherein the additional images include one or more stereoscopic images, each stereoscopic image having a horizontal visual coverage of less than 180 degrees of the room, wherein analyzing the visual data and analyzing the additional visual data are performed without using any depth information about the distance from the position where the one or more images and the additional images are captured to the surrounding surfaces from any depth sensing sensors, wherein the additional visual data further includes visual information in at least one of a video or a three-dimensional model about at least one of the plurality of objects and / or at least one target attribute, wherein the method further includes obtaining, by the one or more computing systems, additional non-visual data having additional details about at least one object and / or at least one target attribute and including at least one of text data or audio data, and wherein determining the contribution of one or more of the plurality of objects is further at least partially based on an analysis of the additional non-visual data and the at least one of the video or the three-dimensional model.

[0181] A20. A computer-implemented method according to any one of clauses A01 to A19, further comprising determining, by the one or more computing systems and at least partially based on an analysis of the visual data of the one or more images, that the visual data lacks a level of detail that meets a threshold defined for the one or more target attributes of each of the plurality of objects, and wherein providing the instructions is performed in response to determining that the visual data lacks the level of detail.

[0182] A21. A computer-implemented method according to any one of clauses A01 to A20, wherein at least one first trained neural network is used to perform the analysis of the visual data of the one or more images, and wherein at least one second trained neural network is used to perform the analysis of the additional visual data of the additional images.

[0183] A22. A computer-implemented method according to any one of clauses A01 to A21, wherein the automated operation further includes determining, by the one or more computing systems, an assessment of the availability of the room for the indicated purpose, including combining information about the assessment of the contribution of the plurality of objects, and wherein providing information about the determined assessment of each of the plurality of objects further includes displaying, by the one or more computing systems, information about the determined assessment of the availability of the room.

[0184] A23. The computer-implemented method according to clause A22, wherein the additional data includes information about at least one of the shape of the room or the layout of the items in the room, and wherein the evaluation of the availability of the room for the indicated purpose is determined at least in part based on the information about at least one of the shape or layout of the room.

[0185] A24. The computer-implemented method according to any one of clauses A22 to A23, wherein the additional data includes at least one of the type of the room or the label of the room, and wherein the automated operation further includes determining, by the one or more computing systems, the purpose of the indication of the room at least in part based on at least one of the type of the room or the label of the room.

[0186] A25. The computer-implemented method according to any one of clauses A01 to A24, wherein the automated operation further includes:

[0187] obtaining, by the one or more computing systems, information about the labels identifying the plurality of objects; and

[0188] identifying, by the one or more computing systems and based on the labels, a plurality of target attributes of the plurality of objects, the plurality of target attributes including the one or more target attributes of each of the plurality of objects.

[0189] A26. The computer-implemented method according to any one of clauses A01 to A25, wherein the automated operation further includes:

[0190] evaluating, at least in part based on information from the additional data about the one or more target attributes of each of the one or more objects, the contribution of the one or more objects to the availability of the room for the purpose of the indication; and

[0191] providing information about the contribution of the evaluation of the one or more objects.

[0192] A27. The computer-implemented method according to clause A26, wherein the one or more objects include a plurality of objects, and wherein the method further includes:

[0193] determining the evaluation of the availability of the room for the purpose of the indication, including combining information about the current contribution of the evaluation of the plurality of objects; and

[0194] Provide information for an assessment of the determination of the availability of the room such that the information for the assessment of the determination of the availability of the room is displayed in a manner associated with other visual information about the room.

[0195] A28. A computer-implemented method according to any one of clauses A26 to A27, wherein the one or more images include one or more initial images with initial visual data,

[0196] wherein analyzing the visual data of the one or more images is performed using at least one first trained neural network,

[0197] wherein the additional visual data is included in at least one of the one or more additional images or one or more videos or one or more 3D models,

[0198] wherein obtaining the one or more images includes identifying the one or more objects at least in part based on an analysis of the one or more initial images; and

[0199] wherein the automated operation further includes assessing one or more target attributes of each of the one or more objects at least in part based on an analysis of the additional visual data using at least one second trained neural network, and using information from the assessment as part of an evaluation of the contribution of the one or more objects.

[0200] A29. A computer-implemented method according to clause A28, wherein the automated operation further includes analyzing information associated with the additional visual data to verify its match with other information associated with the visual data of the one or more images, including by at least one of the following: comparing at least some of the additional visual data with at least some of the visual data of the one or more images to verify that the additional visual data and the visual data show at least one common object, or comparing first acquisition pose information of the one or more additional images with second acquisition pose information of the one or more images to verify that the first acquisition pose information and the second acquisition pose information match.

[0201] A30. A computer-implemented method according to any one of clauses A01 to A29, wherein the automated operation further comprises obtaining, by the one or more computing systems, additional non-visual data having additional details regarding at least one object and / or at least one target attribute, and wherein the automated operation further comprises assessing, at least in part based on the additional non-visual data, the one or more target attributes of each of the one or more objects, and using the information from the assessment as part of evaluating the contribution of the one or more objects.

[0202] A31. A computer-implemented method, which comprises a plurality of steps to perform an automated operation that implements the techniques substantially as described herein.

[0203] B01. A non-transitory computer-readable medium storing executable software instructions and / or other stored content, the executable software instructions and / or other stored content causing one or more computing systems to perform an automated operation that implements the method according to any one of clauses A01 to A31.

[0204] B02. A non-transitory computer-readable medium storing executable software instructions and / or other stored content, the executable software instructions and / or other stored content causing one or more computing systems to perform an automated operation that implements the techniques substantially as described herein.

[0205] C01. One or more computing systems comprising one or more hardware processors and one or more memories storing instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform an automated operation that implements the method according to any one of clauses A01 to A31.

[0206] C02. One or more computing systems comprising one or more hardware processors and one or more memories storing instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform an automated operation that implements the techniques substantially as described herein.

[0207] D01. A computer program that, when run on a computer, is adapted to perform the method according to any one of clauses A01 to A31.

[0208] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions. It will be further understood that in some embodiments, the functionality provided by the routines discussed above can be provided in an alternative manner, such as being divided among more routines or combined into fewer routines. Similarly, in some embodiments, the routines illustrated may provide more or less functionality than described, such as when other illustrated routines are correspondingly changed to lack or include such functionality, or when the amount of functionality provided is changed. Additionally, although various operations may be illustrated as being performed in a particular manner (e.g., serially or in parallel or synchronously or asynchronously) and / or in a particular order, in other embodiments, the operations may be performed in other orders and in other manners. Any data structures discussed above may also be structured differently, such as by dividing a single data structure into multiple data structures and / or by combining multiple data structures into a single data structure. Similarly, in some embodiments, the data structures illustrated may store more or less information than described, such as when other illustrated data structures are correspondingly changed to lack or include such information, or when the amount or type of information stored is changed.

[0209] In view of the foregoing, it should be understood that although specific embodiments have been described herein for purposes of illustration, various modifications can be made without departing from the spirit and scope of the invention. Accordingly, the invention is not limited except as by the corresponding claims and the elements recited in those claims. Additionally, although certain aspects of the invention may be presented at times in certain claim forms, the inventors contemplate various aspects of the invention in any available claim form. For example, although only some aspects of the invention may be recited as being embodied in a computer-readable storage medium at a particular time, other aspects may be so embodied as well.

Claims

1. A computer-implemented method, comprising: Obtaining, by one or more computing systems, one or more images captured in a room of a house; Analyzing, by the one or more computing systems, visual data of the one or more images to identify a plurality of objects installed in the room and determining, for each of the plurality of objects, a type of the object; Determining, by the one or more computing systems and based on the one or more images and for each of the plurality of objects, one or more target attributes of the object for which additional data is to be captured, including determining that the visual data of the one or more images lacks details sufficient to meet a threshold defining the one or more target attributes of the object; Capturing the additional data for the one or more target attributes of each of the plurality of objects, wherein capturing the additional data includes: obtaining additional visual data in one or more additional images of the plurality of objects; Analyzing, by the one or more computing systems and via use of at least one trained neural network, the additional visual data of the additional images to determine, for each of the plurality of objects, whether the additional data for the one or more target attributes of the object has been captured, wherein determining that the additional data has been captured includes: determining whether the additional visual data has sufficient details to meet a threshold defining the one or more target attributes of the object; Responsive to determining that the additional data has not been captured, recapturing and analyzing further additional visual data and verifying that the additional data has been captured; Responsive to determining that the additional data has been captured, determining, by the one or more computing systems, an assessment of the current availability of the room for an indicated purpose, including assessing, at least in part based on information from the additional data for the one or more target attributes of each of the plurality of objects, the contribution of the plurality of objects to the current availability of the room and including combining information on the assessed contribution of the plurality of objects; and Providing, by the one or more computing systems, information on the determined assessment of the current availability of the room.

2. The computer-implemented method according to claim 1, further comprising: Obtaining, by the one or more computing systems, information on labels identifying the plurality of objects; And wherein determining the one or more target attributes of each of the plurality of objects further includes: identifying, by the one or more computing systems and based on the labels, a plurality of target attributes of the plurality of objects, the plurality of target attributes including the one or more target attributes of each of the plurality of objects.

3. The computer-implemented method according to claim 1, wherein, The additional data includes information about at least one of the shape of the room or the layout of the items in the room, and wherein the evaluation of the current availability of the room for the indicated purpose is determined at least in part based on the information about at least one of the shape of the room or the layout of the items in the room.

4. The computer-implemented method according to claim 1, wherein, The additional data includes at least one of the type of the room or the label of the room, and wherein the method further includes: determining, by the one or more computing systems, the purpose of the indication of the room at least in part based on at least one of the type of the room or the label of the room.

5. The computer-implemented method according to claim 1, wherein, Analyzing the additional visual data of the additional image further includes: determining that the additional data is not captured based on the lack of visual data in the additional visual data, the visual data meeting a defined threshold of at least one target attribute of one of the plurality of objects.

6. The computer-implemented method according to claim 1, wherein, Analyzing the additional visual data of the additional image further includes: determining that one of the additional images in the additional image lacks any of the additional data, and wherein the method further includes: capturing, by the one or more computing systems, another copy of the one additional image including at least some of the additional data, and using the another copy of the additional image as part of evaluating the contribution of the plurality of objects to the current availability of the room.

7. The computer-implemented method according to claim 1, wherein, Analyzing the visual data of the one or more images further includes: determining the positions of the plurality of objects in the one or more images, and wherein capturing the additional data further includes: using the information about the determined positions of the plurality of objects.

8. The computer-implemented method according to claim 1, wherein, Each of the plurality of objects installed in the room is at least one of the following: a lighting fixture, or a sanitary fixture, or built-in furniture, or a built-in structure inside the wall of the room, or an electrical appliance, or a gas-powered appliance, or an oil-powered appliance, or an object that produces fire by burning fuel, or an installed floor, or an installed wall covering, or an installed window covering, or the hardware attached to a door, or the hardware attached to a window, or an installed countertop.

9. The computer-implemented method according to claim 1, wherein, Analyzing the visual data of the one or more images further identifies one or more additional objects in the room that are each furniture or movable items, wherein for each of the one or more additional objects, further performing determining the one or more target attributes, and capturing the additional data, and analyzing the additional visual data, and evaluating the contribution, and wherein the evaluation of the current availability of the room for the indicated purpose is further determined at least in part based on the evaluation of the contribution of the one or more additional objects to the current availability of the room.

10. The computer-implemented method according to claim 1, wherein, The evaluation of the current availability of the room for the indicated purpose is determined at least in part based on combining the information on the contributions of the evaluations of the multiple objects and includes at least one of the following: performing, by the one or more computing systems, a weighted average of the contributions of the evaluations of the multiple objects using weights at least partially based on the types of the multiple objects, or providing the contributions of the evaluations of the multiple objects to an additional trained neural network and receiving, from the additional trained neural network, a determined evaluation of the current availability of the room, and wherein the evaluation of the contributions of the multiple objects to the current availability of the room for the indicated purpose and the determination of the current availability of the room for the indicated purpose are performed by evaluating the multiple objects and the characteristics of the room including functionality.

11. The computer-implemented method according to claim 1, wherein, The one or more images include one or more panoramic images that, when combined, include a 360-degree horizontal visual coverage of the room, wherein the additional images include one or more stereoscopic images, each having a horizontal visual coverage of less than 180 degrees of the room, wherein the additional visual data further includes visual information in at least one of a video or a three-dimensional model of at least one of the multiple objects and / or at least one target attribute, and wherein the method further includes: obtaining, by the one or more computing systems, additional non-visual data having additional details about at least one object and / or at least one target attribute and including at least one of text data or audio data, and wherein the determination of the contributions of one or more of the multiple objects is further partially based on an analysis of the at least one of the video or the three-dimensional model and the additional non-visual data.

12. A non-transitory computer-readable medium storing instructions that cause one or more computing devices to perform automated operations, the automated operations at least including: obtaining, by the one or more computing devices, one or more images captured in a room of a building; determining, by the one or more computing devices and at least in part based on analyzing visual data of the one or more images, multiple objects in the room and determining, for each of the multiple objects, the type of the object; capturing, by the one or more computing devices, additional data on one or more target attributes of each of the multiple objects, wherein the one or more target attributes of each of the multiple objects are determined at least in part based on the type of the object, and wherein capturing the additional data includes: obtaining additional visual data in additional images of the multiple objects; Analyze the additional visual data of the additional image via the one or more computing devices to determine, for each of the plurality of objects, whether the additional data regarding one or more target attributes of the object has been captured, wherein determining that the additional data has been captured for each of the plurality of objects includes: determining whether the additional visual data has sufficient detail to meet a defined threshold regarding the one or more target attributes of the object, and wherein analyzing the additional visual data further includes: for one of the plurality of objects, determining that the additional visual data has not been captured by lack of visual data having sufficient detail to meet a defined threshold regarding at least one target attribute of one of the plurality of objects; For the one object, in response to determining that the visual data has not been captured, capture, via the one or more computing devices, another additional image of additional visual data having sufficient detail to meet the defined threshold regarding the at least one target attribute of the one object; For all of the plurality of objects, in response to determining that the additional data has been captured, determine, via the one or more computing devices and for each of the plurality of objects, an assessment of the contribution of the object to the usability of the room for the intended purpose, at least in part based on information from the additional data regarding the one or more target attributes of the object; and Provide, via the one or more computing devices, information regarding the determined assessment for each of the plurality of objects.

13. A system, comprising: One or more hardware processors of one or more computing systems; And One or more memories storing instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing systems to perform automated operations, the automated operations at least including: Obtain one or more images captured in a room of a building; Determine one or more objects in the room, at least in part based on analyzing visual data of the one or more images, and for each of the plurality of objects, determine the type of the object, and at least in part based on the type of the object, determine one or more target attributes of the object, and at least in part based on the visual data of the one or more images lacking sufficient detail to meet a defined threshold regarding those target attributes, determine one or more target attributes for which additional data is to be captured; Capture the additional data regarding the one or more target attributes of each of the one or more objects, including obtaining additional visual data regarding the one or more objects; For each of the one or more objects, analyze the additional visual data as part of determining that the captured additional data has sufficient detail to meet a defined threshold regarding the one or more target attributes of the object; and For each of the one or more objects, provide information on details for determining that the additional data captured has a threshold sufficient to meet the defined threshold for the one or more target attributes of the object.

Images