Automated usability assessment of buildings using visual data from captured room images

Abstract

Techniques are described for automated operations related to analyzing visual data from images captured in rooms of a building, and optionally additional captured data about the rooms, to assess the room layout and other usability information of the rooms of the building, and optionally the entire building, and subsequently using the assessed usability information in one or more further automated ways, such as to improve navigation of the building. The automated operations may include identifying one or more objects to be assessed in each of the rooms, assessing one or more target attributes of each object, assessing the usability of each object using the assessments of the target attributes and assessing the usability of each room relative to a indicated purpose using the assessments of the objects and other room information, and combining the assessments of multiple rooms in the building with other building information to assess the usability of the building relative to its indicated purpose.

Description

Technical Field

[0001] The following disclosure generally relates to techniques for automatically analyzing visual data from images captured in rooms of a building to assess the usability of the rooms in the building and subsequently using the assessed usability information in one or more ways, such as to determine the room layout and identify information about built-in elements of the room, and using that information to assess the usability of the room, and using the assessed room layout and other usability information to improve navigation and other uses of the building. Background Art

[0002] In various fields and situations, such as building analysis, property inventory, real estate acquisition and development, renovation and remodeling services, general contracting, and other situations, it may be desirable to view information about the interior of a house, office, or other building without having to physically visit or enter the building, including determining actual as-built information about the building, rather than design information obtained before the building was constructed. However, it can be difficult to effectively capture, represent, and use such building interior information, including displaying visual information captured within the building interior to a user at a remote location (e.g., enabling the user to fully understand the layout and other details of the interior, including controlling the display in a manner selected by the user). Additionally, while a floor plan of a building can provide some information about the layout and other details of the building interior, in some cases, such use of a floor plan has several disadvantages, including that the floor plan can be difficult to construct and maintain, difficult to accurately scale and populate with information about the interior of a room, difficult to visualize and otherwise use, and the like. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1A to Figure 1B is a diagram depicting an exemplary building environment and one or more computing systems for use in embodiments of the present disclosure, such as for performing automated operations to capture images in a room and subsequently analyzing visual data of the captured images in one or more ways to produce derived information about the room and the building.

[0004] Figures 2A to 2X Examples are shown of automated operations that capture images of a room and then analyze visual data of the captured images in one or more ways, such as for generating and presenting information about floor plans of a building and for assessing room layout and other usability of rooms of the building.

[0005] Figure 3 is a block diagram illustrating a computing system for executing an embodiment of one or more systems that perform at least some of the techniques described in this disclosure.

[0006] Figure 4An example flow diagram of an image capture and analysis (ICA) system routine is shown, according to an embodiment of the present disclosure.

[0007] Figures 5A to 5C An example flow chart for a Mapping Information Generation Manager (MIGM) system routine according to an embodiment of the present disclosure is shown.

[0008] Figures 6A to 6B An example flow diagram of a Building Usability Assessment Manager (BUAM) system routine is shown according to an embodiment of the present disclosure.

[0009] Figure 7 An example flow diagram of a building map viewer system routine is shown according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0010] This disclosure describes a technique for using a computing device to perform automated operations related to analyzing visual data from images captured in rooms of a building to assess the room layout and other usability information for the building's rooms, and optionally for the entire building, and subsequently using the assessed usability information in one or more further automated ways. The images may, for example, include panoramic images (e.g., in equirectangular or other spherical formats) and / or other types of images (e.g., in rectilinear stereo formats) captured from a capture location in or around a multi-room building (e.g., a house, office, etc.), sometimes referred to herein as "target images." Furthermore, in at least some embodiments, the automated operations are performed without requiring or utilizing information from any depth sensor or other distance measurement device regarding the distance from the image capture location to walls or other objects in the surrounding building. In various embodiments, the assessed room layout and other usability information for one or more rooms of the building captured may be further used in various ways, such as in conjunction with generating or annotating a corresponding building floor plan and / or other generated information about the building, including for controlling a mobile device (e.g., an autonomous vehicle) based on structural elements of the room, for display or other presentation in a corresponding GUI (graphical user interface) on one or more client devices via one or more computer networks, and the like. The following includes additional details regarding the automated determination and use of room and building availability information, and in at least some embodiments, some or all of the techniques described herein may be performed via automated operation of a Building Usability Assessment Manager (“BUAM”) system, as discussed further below.

[0011] As described above, automated operation of the BUAM system may include analyzing visual data from the visual coverage of target images captured in one or more rooms of a building for subsequent use in assessing the usability of the one or more rooms and, in some cases, the entire building. In at least some embodiments, one or more initial target images captured in a room are analyzed to identify various types of information about the room, such as to analyze the one or more initial images having room-level visual coverage and identify specific objects in the room or other elements of the room for which additional data is to be captured, including capturing additional target images that provide more detail about the identified objects, including specific target attributes of interest about the objects, in some embodiments, the one or more initial target images may provide a wide angle and include a total of up to 360° horizontal coverage of the room about a vertical axis and between 180° and 360° coverage about a horizontal axis (e.g., one or more panoramic images, such as in a spherical format), and in some embodiments, the additional target images may be more focused than the initial images (e.g., stereo images, such as in a rectilinear format). After capturing those additional images, the automated operation of the BUAM system may further include branching off the additional visual data in the additional visual range of the additional images to obtain sufficient data (e.g., above a defined detail threshold, or otherwise meeting one or more defined detail criteria) to allow for an assessment of one or more target attributes of interest for each of the identified objects. Once an assessment of the target attributes of the identified objects is available, the automated operation of the BUAM system may further include performing an assessment of each of those identified objects based, at least in part, on one or more assessments of the one or more target attributes of the objects, such as to estimate how the object contributes to an overall assessment of the room (e.g., an assessment of the room's usability for the indicated purpose). The automated operation of the BUAM system may further include performing an overall assessment of the room based, at least in part, on a combination of assessments of the identified objects in the room, optionally in combination with other information about the room (e.g., the room's layout, the room's traffic flow, etc.). Similarly, in at least some embodiments, the automated operation of the BUAM system may further include performing an overall assessment of the building based, at least in part, on a combination of assessments of some or all of the rooms in the building, optionally in combination with other information about the rooms (e.g., the building's layout, the building's traffic flow, etc.). Additionally, in at least some embodiments, for the purposes of the analysis discussed herein, areas outside a building may be considered rooms, such as for defined areas (e.g., patios, decks, gardens, etc.) and / or for all surrounding areas (e.g., the outer perimeter of a building), and including identifying and evaluating the usability of objects in such “rooms,” and assessing target attributes of such objects, as well as evaluating the overall usability of the “rooms,” and including the usability of such “rooms” as part of an assessment of the overall usability of the building.

[0012] As described above, in some embodiments, automated operation of the BUAM system may include analyzing one or more initial images captured in a room, including room-level visual coverage, to identify various types of information about the room. Non-exclusive examples of information that may be automatically determined based on the analysis of visual data in the one or more initial images include one or more of the following:

[0013] - the presence of certain visible, identifiable objects in the room or other elements of the room for which additional data is to be captured, such as built-in or otherwise non-transitory mounted objects and / or other objects that are more transient (e.g., easily movable, such as furniture or other decorations, etc.);

[0014] - the presence of specific visible target attributes of a specifically identified object;

[0015] - The location of some or all identified objects in the room (e.g., location within a particular image, such as using a bounding box and / or pixel-level mask; relative to the shape of the room, such as relative to a wall and / or floor and / or ceiling; relative to a geographic direction, such as a west wall or southwest corner, etc.);

[0016] - the types of some or all identified objects in the room, such as using object labels or other categories of objects (e.g., window, door, chair, etc.);

[0017] - The type of room, such as using a room tag or other category of rooms (e.g., bedroom, bathroom, living room, etc.);

[0018] - the layout of the room (e.g., the arrangement of furniture or other items in the room, optionally with respect to the shape of the walls and other structural elements of the room);

[0019] - expected and / or actual traffic flow patterns in the room (e.g., movement between doors and other wall openings of the room, and optionally to one or more other identified areas of the room, such as relative to information about the room layout);

[0020] - the intended purpose of the room (e.g., the type of functionality of the room, such as based on room type and / or layout, including the non-exclusive examples of "kitchen" or "living room," or "cooking" or "personal cooking" or "industrial cooking" for a kitchen, or "group entertainment" or "personal relaxation" for a living room, etc.; and / or the quality of the room and its contents when installed or otherwise new, such as for room types where quality may affect or otherwise be part of some or all of the intended purposes; and / or the condition of the room and its contents at the current time, such as relative to state of repair or disrepair, etc.); and / or the usability of the room, such as based on room layout and / or traffic flow and / or functionality and / or quality and / or condition;

[0021] - Other properties of the room, such as the degree of "openness" and / or the complexity of the room's shape (e.g., cubic, L-shaped, etc.) and / or the degree of accessibility, etc.

[0022] In other embodiments, some or all of the above-described information types may not be used for room availability assessments for some or all rooms and / or building availability assessments for some or all buildings, and / or may be obtained in other ways (e.g., supplied by one or more users, such as system operator users of the BUAM system who participate in data capture in the rooms).

[0023] As described above, each room may have one or more elements or other objects identified as being of interest (e.g., to aid in plan evaluation of the room), such as based on the type of room. Identified objects in a room may include built-in or otherwise installed objects, with non-exclusive examples including the following: windows and / or window hardware (e.g., latch mechanisms, open / close mechanisms, etc.); doors and / or door hardware (e.g., hinges, door handles, locking mechanisms; peepholes, etc.); installed flooring (e.g., tile, wood, laminate, carpet, etc.); installed countertops and / or backsplashes (e.g., on a kitchen island, kitchen or bathroom counter, etc.); installed wall coverings (e.g., wallpaper, paint, wainscoting, etc.); other built-in structures inside a kitchen island or the front wall of a room (e.g., a recessed or raised subset of the flooring, a coffered ceiling, etc.); electrical appliances The invention also includes appliances such as gas-powered appliances or other types of powered appliances (e.g., stoves, ovens, microwave ovens, trash compactors, refrigerators, etc.); light fixtures (e.g., attached to a wall or ceiling); plumbing fixtures (e.g., sinks; bathtubs; showers; toilets; hardware on or in sinks or bathtubs or showers, such as drains, spouts, showerheads, faucets, and other controls, etc.); built-in furniture (e.g., bookshelves, bay window seating areas, etc.); security systems; built-in vacuum systems; air heating and / or cooling systems; water heating systems; various types of pipes or other conduits; various types of electrical wiring; various types of communication wiring; built-in speakers or other built-in electronic devices, etc. In addition, objects in a room may include movable or otherwise temporary objects, non-exclusive examples of which include the following: furniture; decorations such as pictures or curtains, etc. In some embodiments, some or all of the recognized objects for a room may be automatically determined based on analysis of visual data from one or more initial images, while in other embodiments, some or all of the recognized objects for a room may be automatically determined based on the type of room (e.g., for a bathroom, a sink and toilet; for a kitchen, a sink and oven, etc.), such as based on a predefined list of room types.

[0024] In addition, each identified object may have one or more target attributes identified as being of interest (e.g., to aid in planning evaluation of the object), such as based on the type of object and / or the type of room in which the object is located, such target attributes may include physical features and / or sub-elements of the object, and / or may include various types of functionality or other characteristics of the object. Non-exclusive examples of target attributes include the following: size; material; age; installation type; for door objects, one or more pieces of door hardware, a view of the surroundings on the other side, or other indications; for window objects, one or more pieces of window hardware, a view of the surroundings on the other side, or other indications; for sinks or bathtubs or showers, the hardware on or inside thereof, type (e.g., arched foot bathtub, wall-mounted sink, etc.), functionality (e.g., for bathtubs, including jets), model number, etc.; for stoves, the number of burners, the type of energy used (e.g., electric, gas, etc.), model number, other features or functionality such as a built-in fan, etc.; for other appliances, model number or other type, etc. In some embodiments, some or all target attributes of interest for an identified object may be automatically determined based on analysis of visual data from one or more initial images, while in other embodiments, some or all target attributes of interest for an object may be automatically determined based on the type of object and / or the type of room in which the object is located (e.g., for a sink in a bathroom, the sink hardware and the type of sink, such as wall-mounted or freestanding; for a sink in a kitchen, the size of the sink and the number of bowls and the sink hardware; for a sink in a utility room, the size of the sink and the number of bowls), such as based on a predefined list of object types and / or room types.

[0025] In various embodiments, the initial image of the room and additional data about the room can be captured in various ways. In some embodiments, some or all of the initial images of the room can be provided to the BUAM system by another system that has already captured those images for other purposes, such as by an image capture and analysis (ICA) system and / or a mapping information generation manager (MIGM) system that uses images of the rooms of a building to generate floor plans and / or other mapping information related to the building, as discussed in more detail below. In other embodiments, some or all of the initial images of the room can be captured by the BUAM system or in response to instructions provided by the BUAM system, such as to an automated image capture device in the room and / or a user in the room (e.g., a BUAM system operator user), where information indicates the type of initial image to be captured. In a similar manner, in at least some embodiments, some or all of the additional images of the room can be captured by the BUAM system or in response to instructions provided by the BUAM system, such as to an automated image capture device in the room and / or a user in the room (e.g., a BUAM system operator user), where information indicates the type of additional images to be captured. For example, the BUAM system may provide instructions that identify one or more objects of interest in a room for which additional data is to be captured, and identify one or more target attributes for each of the objects of interest for which additional data is to be captured that meets defined detail criteria or otherwise meets one or more defined detail criteria (or otherwise provides a description of the additional data to be captured for the object that results in the capture of sufficient data regarding the one or more target attributes). In various embodiments, such instructions may be provided in various ways, including being displayed to the user in a GUI of the BUAM system on a user's mobile computing device (e.g., a mobile computing device that acts as an image acquisition device and is used to capture some or all of the additional images, such as using one or more imaging sensors of the device and optionally additional hardware components of the device, such as a light, one or more IMU (internal measurement unit) sensors, such as one or more gyroscopes and / or accelerometers and / or magnetometers or other compasses, etc.) or otherwise provided to the user (e.g., superimposed on an image of the room displayed on such computing device and / or other separate camera devices, such as providing dynamic augmented reality instructions to the user as the image changes in response to movement of the device, and / or providing static instructions to the user on a previously captured image, and optionally with visual markers on one or more images of visible objects and / or target attributes), or instead provided to an automated device that acquires the additional images in response to the instructions.

[0026] Additionally, automated operations of the BUAM system may further include analyzing visual data of the images to verify that they include sufficient detail to satisfy a defined detail threshold or otherwise satisfy one or more defined detail criteria, and initiating further automated operations in response to the verification activity. For example, visual data of one or more initial images may be analyzed to determine, for each of some or all objects of interest, whether the one or more initial images already include sufficient data regarding detail of one or more target attributes of interest for the object, and if so, then the assessment of the object and its one or more target attributes may be performed using visual data of the one or more initial images rather than additional images that would otherwise be captured and used, but if not, then the BUAM system may initiate capture of one or more additional images to provide sufficient data regarding detail of the one or more target attributes of interest for the object. Additionally, once one or more additional images have been captured for an object (e.g., additional images of each of one or more target attributes of the object), additional visual data of the one or more additional images may be similarly analyzed to verify that they include visual data of the one or more target attributes with sufficient detail to satisfy a defined detail threshold or otherwise satisfy one or more defined detail criteria, and in some embodiments, to verify that the one or more additional images actually belong to the correct object and / or the correct target attributes (e.g., by comparing the visual terms of the additional images with corresponding visual data of one or more initial images of the object), and if so, those additional images may be verified as usable for assessment of those target attributes and associated evaluation of the object, but if not, the BUAM system may initiate recapture of one or more new additional images that replace the one or more previous additional images with the verification issue and that provide sufficient detail of the one or more target attributes of interest of the object (or initiate other corrective actions, such as requesting additional details about the one or more target attributes in text or other form). In various embodiments, the defined detail threshold or other one or more defined detail criteria may take various forms, such as a minimum number of pixels or other measure of resolution in an image that displays a target attribute or object that is the subject of the image, a minimum lighting level, a maximum amount of blur or other measure of clarity of visual data, and the like.

[0027] Additionally, in at least some embodiments, the automated operation of the BUAM system to capture additional data may also include capturing other types of data in addition to the additional images, either in addition to or in lieu of corresponding additional images, and such as in response to corresponding instructions provided by the BUAM system to a user and / or an automated device. For example, the BUAM system may provide instructions to an image acquisition device (e.g., a mobile computing device having one or more imaging sensors and other hardware components) to capture data regarding attributes of a particular object and / or target using other sensors (e.g., an IMU sensor, a microphone, a GPS or other location sensor, etc.), and provide the other captured additional data to the BUAM system for further analysis, and / or may provide instructions to a user to obtain and provide additional data in forms other than visual data (e.g., textual responses to questions and / or recorded voice responses), such as aspects of object and / or target attributes that may not be readily determined from visual data (e.g., material, age, size, precise location, mounting technique / type, model, one or more types of functionality, etc.). This type of captured additional data in addition to the captured additional images can be used in various ways, including as part of an assessment of a target attribute and its associated evaluation of an object, as discussed in greater detail elsewhere herein. In various embodiments, the details of interest to be obtained for a target attribute and / or object, and the associated instructions provided by the BUAM system, can be automatically determined by the BUAM system in various ways, such as based on a predefined list or other description of the type of target attribute and / or object type and / or room type in which the object and its one or more target attributes are located.

[0028] After the BUAM system has obtained captured additional data of objects of interest and target attributes in the room (whether visual data from captured additional images, other captured additional data, or visual data from one or more initial room-level images), the additional data can be analyzed in various ways to assess each of the target attributes and evaluate each of the objects, such as based at least in part on an assessment of one or more target attributes of the objects. In at least some embodiments, each of the target attributes may be rated by the BUAM system relative to one or more defined rating criteria, such as in a manner specific to the type of target attribute, for example, the target attribute may be rated relative to one or more factors, where non-exclusive examples of such factors include the following: material; age; size; precise location; installation technique / type; model; one or more types of functionality; the quality of the target attribute when installed or when new; the condition of the target attribute at the current time, such as relative to a state of repair or disrepair, etc., as well as factors specific to particular types of objects and / or target attributes (e.g., for doors and / or door locks and / or window latches, the degree of strength and / or other anti-intrusion protection; for doors and / or door handles and / or windows, the degree of decorative appeal, etc.), and if multiple factors are rated individually, in at least some embodiments, the overall rating of the target attribute may be further determined, such as via a weighted average or other combination technique, and optionally where the weights vary based on particular factors. It should be understood that in some embodiments, ratings of particular target attributes of an object relative to particular factors may be provided by one or more users and used in conjunction with other automatically determined evaluations of other target attributes of the object relative to evaluations of the object.

[0029] Additionally, after assessing one or more target attributes of an object, an assessment of the object may be automatically determined by the BUAM system relative to one or more object assessment criteria, whether the same as or different from the assessment criteria for the one or more target attributes of the object. In at least some embodiments, the objects in the room are assessed at least in part relative to the usability of the room for the intended purpose of the room, such as to estimate the object's contribution to the usability of meeting the intended purpose of the room. As a non-exclusive example, a sink in a master bathroom that is of high quality, condition, and functionality (e.g., based at least in part on the sink fixtures or other sink hardware) may contribute significantly to the assessment of the master bathroom and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the master bathroom, such as where a luxurious environment is part of that intended purpose), but may contribute little or no (or even negatively) to the assessment of a utility room and its intended purpose if a bathroom is present (e.g., relative to the overall quality and / or condition and / or functionality of the utility room, such as based on utilitarian functionality as part of that intended purpose). More generally, each of the objects may be evaluated by the BUAM system based at least in part on a combination of one or more assessments of one or more target attributes of the object and, optionally, relative to one or more additional defined object assessment criteria, such as in a manner specific to the type of object. For example, the object may be assessed relative to one or more factors, where non-exclusive examples of such factors include: material; age; size; precise location; installation technique / type; model; one or more types of functionality; the quality of the object when installed or when new; the condition of the object at the current time, such as relative to a state of repair or disrepair, etc., as well as factors specific to particular types of objects and / or target attributes (e.g., for doors, the degree of strength and / or other anti-intrusion protection, decorative appeal, etc.). If multiple factors are assessed individually, in at least some embodiments, an overall assessment of the object may be further determined, such as via a weighted average or other combination technique, and optionally where the weights vary based on particular factors. It should be understood that in some embodiments, an evaluation of a particular object in a room with respect to particular factors, or more generally with respect to usability for the intended purpose of the room, may be provided by one or more users and used in conjunction with other automatically determined evaluations of other objects in the room as part of the evaluation of the room. As described above, the intended purpose of a room may be based at least in part on the type of room, and the usability of the room for the intended purpose may be based on one or more factors, such as functionality, quality, condition, etc.

[0030] Additionally, after evaluating one or more objects of interest in a room, an evaluation of the room may be automatically determined by the BUAM system relative to one or more room evaluation criteria (whether the same as or different from the one or more object evaluation criteria of the room's objects), and in at least some embodiments, the room may be evaluated at least in part relative to usability for the room's intended purpose, such as based at least in part on the evaluation of the one or more objects of interest in the room and their estimated contribution to usability in meeting the room's intended purpose. As a non-exclusive example, a sink in a master bathroom that is of high quality, condition, and functionality (e.g., based at least in part on the sink fixtures or other sink hardware) may contribute significantly to the evaluation of the master bathroom and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the master bathroom, such as based on a luxurious ambiance as part of that intended purpose), but may contribute little or nothing (or even negatively) to the evaluation of a utility room and its intended purpose, if a bathroom is present (e.g., relative to the overall quality and / or condition and / or functionality of the utility room, such as based on utilitarian functionality as part of that intended purpose). As another non-exclusive example, the evaluation of a room may be based at least in part on the compatibility of fixtures and / or other objects within the room, such as to share the same style, quality, etc. More generally, each of one or more rooms in a building may be evaluated by the BUAM system based at least in part on combining one or more evaluations of one or more objects of interest in the room and optionally relative to one or more additional defined room evaluation criteria, such as in a manner specific to the type of room. For example, a room may be evaluated relative to one or more factors, wherein non-exclusive examples of such factors include the following: size; layout; shape; traffic flow; materials; age; the quality of the room when installed or when new; the condition of the room at the current time, such as relative to a state of repair or disrepair, etc., and factors specific to a particular type of room (e.g., for a master bathroom or kitchen, the degree of luxury and / or quality; for a utility room or hallway, the degree of functionality or usability, etc.). If multiple factors are evaluated separately, in at least some embodiments, an overall evaluation of the room may be further determined, such as via a weighted average or other combination technique, and optionally wherein the weights vary based on particular factors. It should be understood that in some embodiments, an assessment of a particular room relative to particular factors, or more generally relative to the intended purpose of the room, may be provided by one or more users and used in conjunction with other automatically determined assessments of other rooms in the same building as part of an overall assessment of the building.

[0031] Additionally, after evaluating the rooms of a multi-room building, an overall evaluation of the building may be automatically determined by the BUAM system relative to one or more building evaluation criteria (whether the same as or different from the room evaluation criteria for the rooms of the building), in at least some embodiments, the building is evaluated at least in part relative to usability for the building's intended purpose, such as based at least in part on an evaluation of the rooms in the building and their assessed satisfaction of their individual intended purposes, optionally in combination with additional factors such as the overall layout of the building and / or the expected traffic flow through the building. As a non-exclusive example, a sink in a bathroom that is of high quality, condition, and functionality (e.g., based at least in part on the sink fixtures or other sink hardware) may contribute significantly to the overall evaluation of the building and its intended purpose (e.g., relative to the overall quality and / or condition and / or functionality of the house) if the building is a single-family home, but may contribute little or no (or even negatively) to the overall evaluation of the building and its intended purpose if the building is a warehouse (e.g., relative to the overall quality and / or condition and / or functionality of the warehouse, such as based on utility as part of that intended purpose). More generally, a building may be evaluated by the BUAM system at least in part based on combining one or more assessments of some or all of the rooms in the building and, optionally, relative to one or more additional defined building assessment criteria, such as in a manner specific to the type of building. For example, a building may be assessed relative to one or more factors, where non-exclusive examples of such factors include: size; layout (e.g., based on the building's floor plan); shape; traffic flow; materials; age; the quality of the building when installed or when new; the condition of the building at the current time, such as relative to a state of repair or disrepair, etc., as well as factors specific to particular types of buildings (e.g., for a house or office building, the degree of luxury and / or quality; for a warehouse or storage facility, the degree of functionality or usability, etc.). If multiple factors are assessed, in at least some embodiments, an overall assessment of the building may be further determined, such as via a weighted average or other combination technique, and optionally where the weights vary based on particular factors. It should be understood that in some embodiments, an assessment of a particular building with respect to particular factors, or more generally with respect to its usability for its intended purpose, may be provided by one or more users and used in conjunction with other automatically determined assessments of other related buildings in a group as part of an overall assessment of the group of buildings.

[0032] As described above, with respect to information from the captured additional data used to assess target attributes and / or evaluate objects and / or evaluate rooms, some or all of this information may be based on analysis of visual data in one or more initial room-level images and / or one or more additional images. As part of automated operation of the BUAM system, in at least some embodiments, the described techniques may include using one or more trained neural networks or other techniques to analyze visual data of the one or more initial images and / or additional images. As non-exclusive examples, such techniques may include one or more of the following: using a trained neural network or other analytical technique (e.g., a convolutional neural network) to take as input an image of some or all of a room and identify objects of interest in the room, such objects may include, for example, wall structural elements (e.g., windows and / or skylights; passages into and / or out of the room, such as doorways and other openings in walls, stairs, corridors, etc.; boundaries between adjacent walls; boundaries between a wall and a floor; boundaries between a wall and a ceiling; corners (or solid geometric vertices) where at least three surfaces or planes intersect, etc.), other fixed structural elements (e.g., countertops, bathtubs, sinks, islands, fireplaces, etc.); using a trained neural network or other analytical technique to take as input one or more images of some or all of a room and determine the room shape of the room, such as a 3D point cloud (where a plurality of 3D data points correspond to locations on the walls and optionally the floor and / or ceiling) and / or fully or partially connected planar surfaces (corresponding to some or all of the walls and optionally the floor and / or ceiling) and / or wireframe structural lines (e.g., to show boundaries between walls, boundaries between a wall and a ceiling, boundaries between a wall and a floor, etc.); using a trained neural network or other analytical technique (e.g., a deep learning detector model or other type of classifier) ​​to take as input one or more images of some or all of the room (and optionally the determined room shape of the room) and determine the positions of detected objects and other elements in the room (e.g., relative to the shape of the room, based on performing object detection to generate bounding boxes around elements or other objects in the one or more images, based on performing object segmentation to generate pixel-level masks identifying pixels representing elements or other objects in the one or more images, etc.); using a trained neural network or other analytical technique (e.g., a convolutional neural network) to take as input one or more images of some or all of the room and determine object labels and / or object types (e.g., windows, doorways, etc.) for those elements or other objects; using a trained neural network or other analytical technique to take as input one or more images of some or all of the room and determine the room type and / or room label (e.g., living room, bedroom, bathroom, kitchen, etc.) of the enclosed room;using a trained neural network or other analytical technique to take as input one or more images of some or all of a room (e.g., a panoramic image with 360° horizontal visual coverage) and determine the layout of the room; using a trained neural network or other analytical technique to take as input one or more images of some or all of a room and determine the expected traffic flow of the room; using a trained neural network or other analytical technique to take as input one or more images of some or all of a room (and optionally information about the room type / label and / or layout and / or traffic flow) and determine the intended purpose of enclosing the room; using a trained neural network or other analytical technique to take as input one or more images of some or all of a room and identify visible target attributes of objects of interest; using a trained neural network or other analytical technique to take as input one or more images of some or all of a room and determine whether one or more visible target attributes have sufficient detail in the visual data to satisfy a defined detail threshold or otherwise satisfy one or more defined detail criteria; using a trained neural network or other analytical technique to take as input one or more images of some or all of a room and determine whether one or more visible objects have sufficient detail in the visual data to satisfy a defined detail threshold or otherwise satisfy a defined detail criteria using a trained neural network or other analytical technique to take as input one or more images of some or all of the object (and optionally additional captured data regarding one or more target attributes and / or the object) and assess each of the one or more target attributes based at least in part on the visual data of the one or more images; using a trained neural network or other analytical technique to take as input one or more images of the object (and optionally additional captured data regarding the object and / or its room, including identifying an intended purpose of the room) and assess the object based at least in part on the visual data of the one or more images; using a trained neural network or other analytical technique to take as input one or more images of the object (and optionally additional captured data regarding the object and / or its room, including identifying an intended purpose of the room) and assess the object based at least in part on the visual data of the one or more images; using a trained neural network or other analytical technique (e.g., using rule-based decision making, such as where predefined rules are specified or otherwise determined by one or more BUAM system operator users) to take as input an assessment of one or more target properties of an object (and optionally additional captured data about the object and / or its room, including identifying an intended purpose of the room) and evaluate the object based at least in part on the assessment of the target properties; using a trained neural network or other analytical technique to take as input an assessment of one or more objects in a room (and optionally additional captured data about the object and / or the room, including identifying an intended purpose of the room) and evaluate the room based at least in part on the assessment of the objects;Using a trained neural network or other analytical technique to take an assessment of one or more rooms in a building (and optionally additional captured data about the rooms and / or the building thereof, including identifying the intended purpose of the building) as input and, based at least in part on the assessment of the rooms, assess the building, etc. In various embodiments, such neural networks may utilize, for example, different detection and / or segmentation frameworks, and in various embodiments, may otherwise be of various types, and may be trained by the BUAM system prior to use on a dataset corresponding to the type of neural network being used. In some embodiments, such image acquisition metadata may further be used to determine one or more of the types of information discussed above, such as using data from an IMU (Internal Measurement Unit) sensor on an acquisition camera or other associated device as part of performing SLAM (Simultaneous Localization and Mapping) and / or SfM (Structure from Motion) and / or MVS (Multi-View Stereo) analysis, or otherwise determining acquired pose information for images of the room, as discussed elsewhere herein.

[0033] The following includes additional details regarding automated operations that, in at least some embodiments, may be performed by the BUAM system to collect and analyze visual data from the visual coverage of target images captured in one or more rooms of a building, and / or use information from the analysis to assess the usability of the rooms. Figures 2P to 2X Examples and their associated descriptions and in Figures 6A to 6B Some corresponding additional details are included in and elsewhere in this document.

[0034] As described above, after assessing the availability of one or more rooms of a building based at least in part on analysis of visual data from images captured in one or more rooms and optionally further assessing the overall availability of the building, the automated operation of the BUAM system may also include using the assessed room and / or building availability information in one or more further automated ways. For example, as discussed in more detail elsewhere herein, such assessment information may be associated with floor plans and / or other generated mapping information for one or more rooms and / or buildings and used to improve automated navigation of a building by a mobile device (e.g., a semi-autonomous or fully autonomous vehicle) based at least in part on the determined assessments of the rooms and buildings (e.g., based on room layout, traffic flow, etc.). In some embodiments, such information regarding room and / or building and / or object assessments and assessments of target attributes of objects may be further used in additional ways, such as displaying information to a user to assist them in navigating the rooms and / or buildings, or for other uses by the user. In some embodiments, such information regarding one or more room and / or building and / or object assessments and ratings regarding the target attributes of the objects may also be used in other ways, such as automatically identifying areas for improvement or renovation in a building (e.g., in specific rooms, and / or with respect to specific objects and / or their target attributes), automatically assessing the price and / or value of a building (e.g., based on comparison with other buildings having similar assessments of overall building usability relative to the building's overall intended purpose and / or having similar assessments of room usability relative to the intended purposes of some or all of the building's rooms), etc. It will be appreciated that in other embodiments, various other uses for the assessment information may be achieved.

[0035] In various embodiments, the described techniques provide various benefits, including allowing floor plans of multi-room buildings and other structures to be automatically enhanced with information regarding assessments of rooms in the building and / or regarding an overall assessment of the building, and optionally assessments of specific objects in the rooms, and assessments of target properties of the building. In some embodiments, such information regarding room and / or building and / or object assessments and assessments of target properties of the objects can also be used in additional ways, such as automatically identifying areas for improvement or renovation in the building (e.g., in specific rooms, and / or with respect to specific objects and / or their target properties), automatically assessing the price and / or value of the building, automatically ensuring that desired types of information are captured and used (e.g., at least in part by associated users who are not experts or otherwise trained in such information capture), etc. Additionally, such automated techniques allow such building, room, and object information to be determined more quickly and, in at least some embodiments, with greater accuracy than previously existing techniques, including by using information gathered from the actual building environment (rather than from plans of how the building should theoretically be built), and by enabling the capture of changes to structural elements or other parts of the building that occur after the building was initially constructed. Such described techniques further provide benefits in allowing improved automated navigation of buildings by mobile devices (e.g., semi-autonomous or fully autonomous vehicles) based at least in part on determined assessments of rooms and buildings (e.g., based on room layout, traffic flow, etc.), including significantly reducing the computing power and time used to attempt to otherwise learn the building layout. Additionally, in some embodiments, the described techniques can be used to provide an improved GUI in which a user can more accurately and quickly obtain information about the interior of a building (e.g., for use in navigating that interior), including in response to search requests, as part of providing personalized information to a user, as part of providing value estimates and / or other information about a building to a user, etc. The described techniques also provide various other benefits, some of which are further described elsewhere herein.

[0036] As described above, in at least some embodiments and scenarios, some or all of the images collected for a building may be panoramic images, each collected at one of a plurality of collection locations in or around the building, such as by generating a panoramic image at each such collection location from one or more of: a video captured at the collection location (e.g., a 360° video shot from a smartphone or other mobile device held by a user and rotated at the collection location), or multiple images captured from the collection location in multiple directions (e.g., from a smartphone or other mobile device held by a user and rotated at the collection location; from automated rotation of the device at the collection location, such as on a tripod at the collection location, etc.), or by simultaneously capturing all image information for a particular collection location (e.g., using one or more fisheye lenses), etc. It should be understood that in some cases such panoramic images may be represented in a spherical coordinate system and provide up to 360° coverage about the horizontal and / or vertical axes (e.g., 360° coverage along the horizontal plane and about the vertical axis), while in other embodiments, the captured panoramic images or other images may include less than 360° horizontal and / or vertical coverage (e.g., for images with typical aspect ratios where the width exceeds the height, such as at or exceeding 21:9 or 16:9 or 3:2 or 7:5 or 4:3 or 5:4 or 1:1, including so-called "ultra-wide" lenses and resulting ultra-wide images). Additionally, it should be understood that a user viewing such a panoramic image (or other image having sufficient horizontal and / or vertical coverage such that only a portion of the image is displayed at any given time) may be permitted to move the viewing direction to different orientations within the panoramic image to indicate that different subsets of the image (or "views") are rendered within the panoramic image, and that in some cases such a panoramic image may be represented in a spherical coordinate system (including, if the panoramic image is represented in a spherical coordinate system and a particular view is rendered, converting the image to a planar coordinate system for rendering, such as for a stereoscopic image view prior to display). Additionally, acquisition metadata regarding the capture of such panoramic images may be obtained and used in various ways, such as data acquired from an IMU (Inertial Measurement Unit) sensor or other sensor of a mobile device as the mobile device is carried by a user or otherwise moved between acquisition locations, non-exclusive examples of such acquisition metadata may include one or more of the following: acquisition time; acquisition location, such as GPS coordinates or other location indication; acquisition direction and / or orientation; relative or absolute acquisition order of multiple images acquired or otherwise associated with a building, etc., and in at least some embodiments and scenarios, such acquisition metadata may also optionally be used as part of determining the acquisition location of the image, as further discussed below. Included below are additional details regarding the automated operations involved in acquiring images and optionally acquiring metadata by one or more devices implementing an image capture and analysis (ICA) system, including information regarding Figure 1A to Figure 1B and Figures 2A to 2D and Figure 4 and additional details elsewhere in this article.

[0037] Also as described above, in various embodiments, the shapes of the rooms of a building may be automatically determined in various ways, including at a time prior to automated determination of the capture location of a particular image within the building. For example, in at least some embodiments, a Mapping Information Generation Manager (MIGM) system may analyze various images captured in and around a building to automatically determine the room shapes (e.g., 3D room shapes, 2D room shapes, etc.) of the rooms of the building and automatically generate floor plans of the building. As an example, if multiple images are captured within a particular room, those images may be analyzed to determine the 3D shape of the rooms of the building (e.g., to reflect the geometry of the surrounding structural elements of the building), the analysis may include, for example, automated operations to "register" the camera positions of the images in a common reference frame in order to "align" the images, and estimate the 3D positions and shapes of objects in the room, such as determining features visible in the content of such images (e.g., to determine the direction and / or orientation of the capture device when the particular image was captured, the path the capture device traveled through the room, etc.), such as using SLAM techniques for multiple video frame images and / or using other SfM techniques for separating at most certain frames of the image. The method may further include determining and aggregating information about the planes of the detected features and the normal (orthogonal) directions of those planes to identify planar surfaces of possible locations of walls and other surfaces of the room and connecting the various possible wall positions (e.g., using one or more constraints, such as having 90° angles between walls and / or between walls and floor, as part of a so-called "Manhattan world assumption") and forming an estimated room shape for the room. After determining the estimated room shapes for the rooms in the building, in at least some embodiments, the automated operations may also include positioning the multiple room shapes together to form a floor plan and / or other related mapping information for the building, such as by connecting the various room shapes. Thus, such a building floor plan may have associated room shape information and, in various embodiments, may have various forms, such as a 2D (two-dimensional) floor map of the building (e.g., an orthographic top view or other overhead view of a schematic floor map that does not include or display height information) and / or a 3D (three-dimensional) or 2.5D (two and a half dimensions) floor map model of the building that displays height information.As part of the automated analysis of visual data of one or more target images, automated operations may include determining an acquisition position and, optionally, an orientation of a target image captured in a room of a house or other building (or in another defined area), and using the determined acquisition position and, optionally, orientation of the target image to further analyze the visual data of the target image, the combination of the acquisition position and orientation of the target image sometimes being referred to herein as the "acquisition pose" or "acquisition position" or simply "pose" or "position" of the target image. Additional details regarding automated operations involved in implementing one or more devices of the MIGM system in determining room shapes and combining room shapes to generate floor plans are included below, including with respect to. Figure 1A to Figure 1B and Figures 2E to 2M as well as Figures 5A to 5C and additional details elsewhere in this article.

[0038] For illustrative purposes, some embodiments are described below in which specific types of information are collected, used, and / or presented for specific types of structures in specific ways and using specific types of devices—however, it will be understood that the described techniques may be used in other ways in other embodiments, and thus the present invention is not limited to the exemplary details provided. As a non-exclusive example, although some examples discuss assessments of specific types of objects and rooms in a house, it should be understood that in other embodiments, other types of assessments may be similarly generated, including for buildings (or other structures or layouts) separate from the house. As another non-exclusive example, although in some examples, specific types of instructions are provided in a specific manner to obtain specific types of data, in other embodiments, other types of instructions may be used and other types of data may be collected in other ways. In addition, the term "building" as used herein refers to any partially or completely enclosed structure, typically but not necessarily including one or more rooms that visually or otherwise divide the interior space of the structure. Non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, condominiums, office buildings, commercial buildings, or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.), etc. As used herein, the terms "collection" or "capture" with reference to a building interior, a collection location, or other location (unless the context clearly indicates otherwise) may refer to any recording, storage, or recording of media, sensor data, and / or other information related to spatial and / or visual and / or otherwise perceptible characteristics of a building interior, or a subset thereof, such as by a recording device or by another device that receives information from a recording device. As used herein, the term "panoramic image" may refer to a visual representation that is based on, includes, or is separable into multiple discrete component images originating from substantially similar physical locations in different directions and depicting a larger field of view than any one of the discrete component images depicts alone, including images from a physical location with a sufficiently wide viewing angle to include angles beyond what is perceptible by a person gazing in a single direction (e.g., greater than 120°, 150°, 180°, etc.). As used herein, the term "series" of collection locations generally refers to two or more collection locations, each of which is visited at least once in a corresponding order, regardless of whether other non-collection locations are visited in between, and regardless of whether the visits to the collection locations occur during a single continuous time period or at multiple different times, or by a single user and / or device or by multiple different users and / or devices. In addition, various details are provided in the drawings and text for illustrative purposes, but these details are not intended to limit the scope of the invention. For example, the sizes and relative positions of elements in the drawings are not necessarily drawn to scale, and some details are omitted and / or provided more prominently (e.g., via size and positioning) to enhance readability and / or clarity. In addition, the same reference numerals may be used in the drawings to identify similar elements or actions.

[0039] Figure 1A is an example block diagram of various computing devices and systems that may participate in the described techniques in some embodiments. In particular, Figure 1A 1 , a panoramic image 165 is shown generated by an internal capture and analysis ("ICA") system 160, executing in this example on one or more server computing systems 180, such as with respect to one or more buildings or other structures, and wherein inter-image directional links have optionally been generated for at least some image pairs. Figure 1B An example of such a panoramic image acquisition location 210 of a particular house 198 is shown (e.g., inter-image relative direction links 215-AB, 215-AC, and 215-BC between pairs of images from acquisition locations 210A and 210B, 210A and 210C, and 210B and 210C, respectively), as discussed further below, and additional details related to automated operation of the ICA system are included elsewhere herein, including with respect to Figure 4 In at least some embodiments, at least some of the ICA system may be executed in part on a mobile computing device 185 (whether in addition to or in place of the ICA system 160 on one or more server computing systems 180), such as within an optional ICA application 154, to control the acquisition of target images and optionally additional non-visual data by the mobile computing device and / or one or more nearby (e.g., in the same room) optional separate camera devices 186 operating cooperatively with the mobile computing device, such as with respect to Figure 1B Further discussion. MIGM (Mapping Information Generation Manager) system 160 is further Figure 1A to generate and provide building floor plans 155 and / or other mapping related information based on the use of the panoramic images 165 and optionally associated metadata regarding their acquisition and linking, Figures 2M to 2O (referred to herein as "2-0" for clarity) shows an example of such a floor plan, as discussed further below, and includes additional details elsewhere herein related to the automated operation of the MIGM system, including with respect to Figures 5A to 5B Additional details.

[0040] Figure 1AFurther shown is a BUAM (Building Usability Assessment Manager) system 140 that executes on one or more server computing systems 180 to automatically analyze visual data from images 144 captured in rooms of a building (e.g., based at least in part on panoramic images 165) to assess the availability of rooms in the building, and subsequently use the assessed availability information in one or more ways, including generating and using various availability information 145 during operation of the BUAM system (e.g., information about objects in the room and their target attributes, images of objects and their target attributes, room layout data, assessments of target attributes, assessments of objects and / or rooms and / or buildings, etc.). In at least some embodiments and scenarios, one or more users of the BUAM client computing devices 105 may further interact with the BUAM system 140 via one or more internets 170, such as to assist in some of the automated operations of the BUAM system. Additional details regarding the automated operations of the BUAM system are included elsewhere herein, including regarding Figures 2P to 2X and Figures 6A to 6B In some embodiments, the ICA system and / or the MIGM system and / or the BUAM system 140 may execute on the same one or more server computing systems, such as if multiple or all of those systems are operated by a single entity or otherwise executed in coordination with one another (e.g., some or all of the functionality in those systems are integrated together into a larger system), while in other embodiments, the BUAM system may operate separately from the ICA and / or MIGM systems (e.g., without using any information generated by the ICA and / or MIGM systems).

[0041] One or more users (not shown) of one or more client computing devices 175 may further interact with the BUAM system 140 and optionally the ICA system and / or the MIGM system via one or more computer networks 170, such as to assist in automated operation of one or more systems and / or obtain information generated by one or more of the systems (e.g., captured images; generated floor plans, such as having information about generated object and / or room and / or building assessments superimposed on or otherwise associated with the floor plans, and / or having information about one or more captured images superimposed on or otherwise associated with the floor plans); information about generated object and / or room and / or building assessments, etc.) and optionally interact therewith, including optionally changing between a view of the floor plan and a view of a particular image at a capture location within or near the floor plan; changing the horizontal and / or vertical viewing direction according to which a corresponding view of a panoramic image is displayed, such as determining the portion of the panoramic image to which the current user viewing direction is directed, etc. Additionally, although Figure 1AAlthough not illustrated in , a floor plan (or portion thereof) may be linked to or otherwise associated with one or more other types of information, including a floor plan of a multi-story or other multi-story building having multiple associated sub-floor plans of different floors or levels that are interconnected (e.g., via connected stairways), a two-dimensional ("2D") floor plan of a building linked to or otherwise associated with a three-dimensional ("3D") rendered floor plan of the building, etc. Additionally, while in Figure 1A Not illustrated, but in some embodiments, client computing device 175 (or other devices, not shown) may receive and use information about generated object and / or room and / or building assessments (optionally in combination with generated floor plans and / or other generated mapping-related information) in additional manners, such as to control or assist in automated navigation activities of such devices (e.g., autonomous vehicles or other devices), either in lieu of or in addition to display of the generated information.

[0042] exist Figure 1A In the depicted computing environment of , network 170 may be one or more publicly accessible linked networks, such as the Internet, that may be operated by various different parties. In other embodiments, network 170 may have other forms. For example, network 170 may instead be a private network, such as a company or university network that is completely or partially inaccessible to non-privileged users. In still other embodiments, network 170 may include both private networks and public networks, wherein one or more of the private networks may access one or more of the public networks and / or access one or more of the private networks from one or more of the public networks. Additionally, in various scenarios, network 170 may include various types of wired and / or wireless networks. Additionally, client computing devices 105 and 175 and server computing system 180 may include various hardware components and stored information, as described below with respect to Figure 3 Discuss in more detail.

[0043] exist Figure 1AIn an example, the ICA system 160 may perform automated operations involving generating a plurality of target panoramic images (e.g., each being a 360-degree panorama about a vertical axis) at a plurality of associated acquisition locations (e.g., in a plurality of rooms or other locations within a building or other structure and, optionally, around some or all of the exterior of the building or other structure), such as for generating and providing a representation of the interior of a building or other structure. In some embodiments, other automated operations of the ICA system may further include: analyzing information to determine a relative position / orientation between each of two or more acquisition locations; creating inter-panorama position / orientation links in the panorama to each of one or more other panoramas based on such determined position / orientation; and then providing information to display or otherwise present the plurality of linked panoramic images of the respective acquisition locations within the building, while in other embodiments, some or all of such other automated operations may instead be performed by the MIGM system.

[0044] Figure 1B Depicted is a block diagram of an exemplary building interior environment in which linked panoramic images have been generated and are ready for use in generating and providing corresponding building floor plans and for presenting the linked panoramic images to a user. In particular, Figure 1BA building 198 (in this example, a house 198) is included that has an interior that is at least partially captured via a plurality of panoramic images, such as by a user (not shown) carrying a mobile device 185 with image capture capabilities and / or one or more separate camera devices 186 traversing the building interior to a series of plurality of capture locations 210. Embodiments of an ICA system (e.g., the ICA system 160 on one or more server computing systems 180; a copy of some or all of the ICA system executing on a user's mobile device, such as the ICA application system 154 executing in memory 152 of the device 185, etc.) may automatically perform or assist in capturing data representing the building interior and, in some embodiments, further analyzing the captured data to generate linked panoramic images to provide a visual representation of the building interior. While the user's mobile device may include various hardware components, such as one or more cameras or other imaging systems 135, one or more sensors 148 (e.g., gyroscopes 148a, accelerometers 148b, compasses 148c, etc., such as part of one or more IMUs or inertial measurement units of the mobile device); one or more hardware processors 132, memory 152, a display 142 (e.g., including a touch-sensitive display screen), optionally one or more depth sensors 136, optionally other hardware elements (e.g., an altimeter; a light detector; a GPS receiver; additional memory or other storage, whether volatile or non-volatile; a microphone; one or more external lights; communication 180; a microphone; one or more external lights; etc.), but in at least some embodiments, the mobile device is unable to access or use equipment (such as depth sensor 136) to measure the depth of objects in the building relative to the location of the mobile computing device, so that the relationship between different panoramic images and their capture locations can be determined based in part or in whole on matching features in different images and / or by using information from other listed hardware components, but without using any data from any such depth sensor. Although not illustrated for the sake of brevity, the one or more camera devices 186 can similarly each include at least one or more image sensors and storage for storing captured target images, as well as transmission capabilities for transmitting the captured target images to other devices (e.g., associated mobile computing device 185, remote server computing system 180, etc.), optionally as well as one or more lenses and lights, and optionally in some embodiments, some or all of the other components of the mobile computing device are shown.Additionally, while direction indicators 109 are provided for viewer reference, in at least some embodiments, the mobile device and / or ICA system may not use such absolute direction information, such as instead determining relative directions and distances between panoramic images 210 without regard to actual geographic location or direction.

[0045] In operation, the mobile computing device 185 and / or camera device 186 (for Figure 1B In one embodiment, the image acquisition device or devices may be carried or otherwise accompanied by one or more users, and in other embodiments and situations, may be mounted on or carried by one or more autonomous vehicles that move through the building under their own power. Additionally, in various embodiments, capturing visual data from the capture location may be performed in various ways (e.g., using one or more lenses that capture all image data simultaneously, an associated user rotating his or her body while holding the one or more image capture devices stationary relative to the user's body, an automated device mounted or carrying the one or more image capture devices rotating the one or more image capture devices, etc.), and may include recording video at the capture location and / or taking a series of one or more images at the capture location, including capturing visual information depicting a number of elements or other objects (e.g., structural details) that may be visible in images (e.g., video frames) captured from or near the capture location. Figure 1B In the example of , such elements or other objects include various elements that are structurally part of the walls of the rooms of the house (or structural "wall elements"), such as doorways 190 and 197 and their doors (e.g., with pivoting doors and / or sliding doors), windows 196, and wall boundaries (e.g., corners or edges) 195 (including corner 195-1 at the northwest corner of the house 198, corner 195-2 at the northeast corner of the first room (living room), and corner 195-3 at the southwest corner of the first room), and in addition, Figure 1BIn the example of , such elements or other objects may further include other elements within the room, such as furniture 191-193 (e.g., sofa 191; chair 192; table 193, etc.), pictures or paintings hanging on the wall, or televisions or other objects 194 (e.g., 194-1 and 194-2), lighting fixtures, etc. The one or more image capture devices further optionally capture additional data (e.g., additional visual data using imaging system 135, additional motion data using sensor module 148, additional depth data optionally using distance measurement sensor 136, etc.) at or near the capture location 210A while rotating, and further optionally capture such additional data as the one or more image capture devices move to and / or from the capture location. In some embodiments, the actions of one or more image acquisition devices may be controlled or facilitated using one or more programs executed on the mobile computing device 185 (e.g., via automated instructions to one or more image acquisition devices or to another mobile device (not shown) that carries those devices through the building under its own power; via instructions to associated users in the room; etc.), such as the ICA application system 154 and / or the optional browser 162, the control system 147 for managing the I / O (input / output) and / or communications and / or networks of the device 185 (e.g., receiving instructions from the user and presenting information to the user), etc. The user may also optionally provide a textual or auditory identifier associated with the acquisition location, such as "entrance" for acquisition location 210A or "living room" for acquisition location 210B, while in other embodiments, the ICA system may automatically generate such identifiers (e.g., by automatically analyzing video and / or other recorded information of the building to perform a corresponding automated determination, such as by using machine learning), or no identifier may be used.

[0046] After the first acquisition location 210A has been appropriately captured, the one or more image acquisition devices (and the user, if present) may proceed to the next acquisition location (such as acquisition location 210B along the travel path 115), optionally recording movement data by the one or more image acquisition devices during movement between the acquisition locations, such as visual data and / or other non-visual data from hardware components (e.g., from one or more IMUs 148, from the imaging system 135 and / or one or more camera devices 186, from the distance measurement sensor 136, etc.). At the next acquisition location, the one or more image acquisition devices may similarly capture one or more target images from that acquisition location, and optionally capture additional data at or near that acquisition location. This process may be repeated for some or all rooms of the building, and optionally also for the exterior of the building, as shown for acquisition locations 210C to 210S. The video and / or other images captured by one or more image capture devices for each capture location are further analyzed to generate a target panoramic image for each of the capture locations 210A to 210S, including, in some embodiments, stitching together multiple component images to create a panoramic image and / or matching objects and other elements in different images.

[0047] In addition to generating such panoramic images, in at least some embodiments, further analysis may be performed by the MIGM system (e.g., concurrently with or subsequent to the image capture activity) to determine the room shape of each room (and optionally for other defined areas, such as a deck or other patio or other exterior defined area outside the building), including optionally determining acquisition location information for each target image, and optionally further determining a floor plan of the building and / or other relevant mapping information for the building (e.g., a set of interconnected linked panoramic images, etc.), e.g., to "link" together at least some of the panoramas and their acquisition locations (wherein example acquisition locations are shown for illustration purposes). The replica of the MIGM system may determine relative position information between pairs of acquisition locations that are visible to each other, store corresponding inter-panoramic links (e.g., links 215-AB, 215-BC, and 215-AC between acquisition locations 210A and 210B, 210B and 210C, and 210A and 210C, respectively), and in some embodiments and scenarios, further link at least one acquisition location that is not visible to each other (e.g., link 215-BE (not shown) between acquisition locations 210B and 210E; link 215-CS (not shown) between acquisition locations 210C and 210S, etc.).

[0048] In addition, the mobile computing device 185 and / or camera device 186 may operate under the control of a BUAM system (whether system 140 on one or more server computing systems 180 or a BUAM application 156 executing in memory 152 of the mobile computing device 185) to capture images of the room and objects in the room and their target attributes, whether in lieu of or in addition to performing image acquisition operations of the ICA system (e.g., in some embodiments, capturing images for both systems simultaneously, capturing images for only the BUAM system and not the ICA system, etc.). In a manner similar to that discussed above with respect to the ICA system, the image acquisition device may be moved through some or all of the rooms of the building 198 to capture an initial image and additional images (e.g., simultaneously, such as if analysis of visual data of the initial image is performed in real time or near real time, such as within seconds or minutes of capturing the initial image; in two or more different passes through the building, such as one or more first passes for capturing the initial image and one or more second passes for capturing additional images, etc.), but in other cases the BUAM system may capture only additional images (e.g., if an image from another system (such as an ICA system) is used as the initial image) and / or capture only the initial image (e.g., if the initial image includes sufficient visual detail regarding all objects and target attributes of the objects to perform an assessment of the target attributes and an evaluation of the objects and an evaluation of the rooms). The acquisition of the initial image and / or additional images by the BUAM system may, for example, include following path 115 in whole or in part through acquisition location 210, and may optionally include deviating from the path to capture sufficient detail regarding various objects and / or object attributes. In at least some instances, one or more image acquisition devices may be carried or otherwise accompanied by one or more users participating in the acquisition of the initial image and / or additional images for the BUAM system, while in other embodiments and scenarios, the image acquisition devices may be mounted on or carried by one or more self-propelled devices that move through the structure under their own power. Furthermore, in various embodiments, the capture of visual data may be performed in various ways, as discussed in more detail above with respect to the operation of the ICA system. The one or more image acquisition devices further acquire additional data for the BUAM system (e.g., additional visual data using imaging system 135, additional motion data using sensor module 148, optionally additional depth data using distance measurement sensor 136, etc.), as well as data input or otherwise provided by one or more accompanying users (e.g., BUAM system operator users).In some embodiments, the actions of one or more image acquisition devices may be controlled or facilitated through the use of one or more programs executing on the mobile computing device 185 (e.g., via automated instructions to the one or more image acquisition devices or to another mobile device (not shown) that carries those devices through the building under its own power; via instructions to associated users in the room; etc.), such as the BUAM application system 156 and / or the BUAM system 140. After capturing the various initial and additional images, as well as any other additional data, the BUAM system continues to perform its automated operations to assess target attributes and evaluate usability information for the object, room, and / or building, and uses the generated usability information in various ways.

[0049] about Figure 1A to Figure 1B Various details are provided but it is understood that the details provided are non-exclusive examples included for illustrative purposes and that other embodiments may be practiced in other ways without some or all of such details.

[0050] Figures 2A to 2X Examples are described of automatically capturing images associated with a building and analyzing visual data of the images (and optionally additional types of captured data) to generate and use various information about the building and its rooms, such as generating floor plans of the building, room shapes of the rooms of the building, assessments of the rooms and objects in the rooms (and target attributes of the objects), and usability assessments of the building, at least some of the images being taken in a manner not described herein. Figure 1B Captured at acquisition location 210 within the building 198 in question.

[0051] In particular, Figure 2A An example image 250a is shown, such as in Figure 1B A non-panoramic stereoscopic image (or a northeast-facing subset view of a 360-degree panoramic image captured from this capture location and formatted in a rectilinear manner) of the living room of house 198 is shown, such as captured by an ICA system and / or a BUAM system as the initial image. A direction indicator 109a is further displayed in this example to illustrate the northeast direction from which the image was captured. In the illustrated example, the displayed image includes built-in elements (e.g., light fixture 130a), furniture (e.g., chair 192-1), two windows 196-1, and a picture 194-1 hanging on the north wall of the living room. Inter-room passageways (e.g., doorways or other wall openings) leading into and out of the living room are not visible in this image. However, multiple room boundaries are visible in image 250a, including a horizontal boundary between the visible portion of the living room's north wall and the living room's ceiling and floor, a horizontal boundary between the visible portion of the living room's east wall and the living room's ceiling and floor, and a vertical inter-wall boundary 195-2 between the north and east walls.

[0052] Figure 2B Continue to show Figure 2A and illustrates an example of an image acquisition device that is Figure 1B An additional stereoscopic image 250b is captured from capture location 210B in a northwest direction of the living room of house 198 (or a northwest-facing subset view of a 360-degree panoramic image captured from that capture location and formatted in a rectilinear manner), further displaying a direction indicator 109b in this example to illustrate the northwest direction from which the image was captured. In this example image, a small portion of one of windows 196-1 continues to be visible, along with a portion of window 196-2 and new lighting fixture 130b. Additionally, the horizontal and vertical room boundaries are aligned with the horizontal and vertical room boundaries. Figure 2A A similar approach is visible in image 250b.

[0053] Figure 2C Continue to show Figures 2A to 2B and shows an example of an image acquisition device in which one or more image acquisition devices are Figure 1B 196-2 and the like. In the example image, a portion of the window 196-2 continues to be visible, as does the sofa 191 and the visual horizontal and vertical room boundaries. Figure 2A and Figure 2B This example image further illustrates a wall opening for entering and exiting the living room, which in this example is a doorway 190-1 ( Figure 1B It is recognized as a door leading to the exterior of the house.) It will be appreciated that a variety of other stereoscopic images may be captured from acquisition location 210B and / or other acquisition locations and displayed in a similar manner.

[0054] Figure 2D Shown Figure 1B Other information 255d about a portion of house 198, including a living room and a limited portion of other rooms to the east of the living room. Figure 1B and Figures 2A to 2CAs discussed, in some embodiments, target panoramic images may be captured at various locations within the interior of a house, such as at locations 210A and 210B in the living room, with the corresponding visual content of one or two such resulting target panoramic images subsequently used to determine the room shape of the living room. Additionally, in at least some embodiments, additional images may be captured, such as where one or more image acquisition devices (not shown) capture video or one or more other series of continuous or nearly continuous images while moving through the interior of the house. In this example, the Figure 1B The present invention also provides information about a portion of the path 115 shown, and in particular, a series of locations 215 along the path where one or more image capture devices may capture (e.g., if capturing video data) one or more video frame images (or other series of continuous or nearly continuous images) of the surrounding interior of a house as the one or more image capture devices move, examples of such locations include capture locations 240a through 240c, with other information being provided from the image capture devices. Figures 2E to 2J In this example, the locations 215 along the path are shown separated by a short distance (e.g., a foot, an inch, a fraction of an inch, etc.), but it will be understood that the video capture may be substantially continuous and, thus, in at least some embodiments, only a subset of such captured video frame images (or other images from a series of continuous or nearly continuous images) may be selected and used for further analysis, such as images separated by a defined distance and / or separated by a defined amount of time between captures (e.g., a second, a fraction of a second, a few seconds, etc.) and / or based on other criteria.

[0055] Figures 2E to 2J Continue to show Figures 2A to 2D , and illustrates analyzing additional information about a living room and about portions of the 360° image frames from video captured along path 155 as a type of estimate of a portion of the possible shape of the room (such as determined by the MIGM system). Although not illustrated in these figures, similar techniques may be performed for target panoramic images captured by the camera device at two or more of the acquisition locations 210A, 210B, and 210C, whether supplemental analysis. Figure 2D Alternatively, the additional image frames shown (e.g., to use the visual data of the target image to generate additional estimates of the likely shape of the room) may be used instead of analyzing Figure 2D Additional image frames are shown. In particular, Figure 2E Information 255e is included that explains that the 360° image frame taken from position 240b will share information about various visible 2D features with the 360° image frame taken from position 240a, but for simplicity only the information about the visible 2D features is shown. Figure 2E A limited subset of such features is shown in Figure 1 for a portion of a living room. Figure 2EIn FIG, example sight lines 228 are shown from position 240b to various example features in the room, and similar example sight lines 227 are shown from position 240a to corresponding features, which illustrates the degree of difference between the perspectives at significantly separated capture locations. Figure 2D A series of images of the location 215 may provide various information about the characteristics of the living room, such as Figures 2F to 2I Further explanation.

[0056] In particular, Figure 2F Information 255f showing the northeast portion of the living room visible in the subset of 360° image frames taken from positions 240a and 240b, and Figure 2G Information 255g of the northwest portion of the living room visible in the other subset of the 360° image frames taken from locations 240a and 240b is shown, where various example features in those portions of the living room are visible in both 360° image frames (e.g., corners 195-1 and 195-2, windows 196-1 and 196-2, etc.). As part of automated analysis of the 360° image frames using SLAM and / or MVS and / or SfM techniques, partial information about planes 286e and 286f corresponding to portions of the north wall of the living room can be determined based on the detected features, and partial information 287e and 285f about portions of the east and west walls of the living room can similarly be determined based on corresponding features identified in the images. In addition to identifying such partial planar information of detected features (e.g., each point in the determined sparse 3D point cloud from image analysis), SLAM and / or MVS and / or SfM techniques may also determine information regarding the likely position and orientation / direction 220 of the subset of images from capture location 240a and the likely position and orientation / direction 222 of the subset of images from capture location 240b (e.g., the positions of capture locations 240a and 240b, respectively). Figure 2F Positions 220g and 222g, and optionally Figure 2F The directions of the image subsets shown are 220e and 222e; and the capture locations 240a and 240b, respectively. Figure 2G corresponding positions 220g and 222g, and optionally Figure 2G The directions of the image subsets shown are 220f and 222f). Although Figure 2F and Figure 2G, only features of a portion of the living room are shown, but it will be understood that other portions of the 360° image frame corresponding to other portions of the living room can be analyzed in a similar manner to determine possible information about possible planes of various walls of the room and other features of the living room (not shown). Additionally, similar analysis can be performed between some or all other images at the location 215 selected for use in the living room, resulting in a variety of determined feature planes that may correspond to portions of the walls of the room from the various image analyses.

[0057] Figure 2H Continue to show Figures 2A to 2G , and shows information 255h from analysis of 360° image frames captured at locations 240a and 240b regarding various determined feature planes that may correspond to portions of the west and north walls of a living room. The plane information shown includes a determined plane 286g near or at the north wall (and therefore the corresponding possible locations of portions of the north wall), and a determined plane 285g near or at the west wall (and therefore the corresponding possible locations of portions of the west wall). As would be expected, there are many variations in the different determined planes of the north and west walls from the different features detected in the analysis of the two 360° image frames, such as differences in position, angle and / or length, and missing data for some portions of the walls, resulting in uncertainty as to the actual precise position and angle of each wall. Although Figure 2H It is not illustrated in FIG, but it is understood that similarly determined feature planes of other walls of the living room will be similarly detected, as well as determined feature planes corresponding to features not along walls (eg, furniture).

[0058] Figure 2I Continue to show Figures 2A to 2H , and shows information 255i regarding additional determined characteristic plane information corresponding to portions of the west and north walls of the living room from analysis of various additional 360° image frames selected from additional locations 215 along the path 115 in the living room, as would be expected, in this example, analysis of the other images provides an even greater variation of different determined planes of the north and west walls. Figure 2I Also shown is additional determined information used to aggregate information about various determined characteristic planar portions to identify possible portion locations 295a and 295b of the west and north walls, as shown. Figure 2J The information 255j is shown. In particular, Figure 2IInformation 291a regarding normal directions for some of the determined feature planes corresponding to the west wall is shown, along with additional information 288a regarding those determined feature planes. In an example embodiment, the determined feature planes are clustered to represent hypothesized wall locations for the west wall, and information regarding the hypothesized wall locations is combined to determine possible wall locations 295a, such as by weighting the information from the various clusters and / or underlying determined feature planes. In at least some embodiments, the hypothesized wall locations and / or normal information are analyzed using machine learning techniques, optionally by further applying assumptions or other constraints (such as 90° corners, as described above) as part of the machine learning analysis. Figure 2H , and / or having flat walls) or the results of the analysis are applied to determine the resulting possible wall locations. A similar analysis may be performed for the north wall using information 288b about corresponding determined characteristic planes and additional information 291b about the resulting normal directions of at least some of those determined characteristic planes. Figure 2J Resulting possible partial wall positions 295a and 295b are shown for the west and north walls, respectively, of the living room, including optionally estimating the positions of missing data (eg, via interpolation and / or extrapolation using other data).

[0059] although Figure 2I It is not shown in FIG, but it should be understood that similarly determined characteristic planes and corresponding standard directions of other walls of the living room will be similarly detected and analyzed to determine their possible positions, thereby resulting in an estimated partial overall room shape of the living room based on visual data collected in the living room by one or more image acquisition devices. Additionally, similar analysis is performed for each room of the building, thereby providing an estimated partial room shape for each room. Additionally, although Figures 2D to 2J Although not illustrated in the drawings, in some embodiments, the analysis of visual data captured by the one or more image capture devices may be supplemented and / or replaced by analysis of depth data (not shown) captured by the one or more image capture devices in the living room, such as to generate an estimated 3D point cloud directly from depth data representing the walls and optionally the ceiling and / or floor of the living room. Figures 2D to 2J Also not described, but in at least some embodiments, other room shape estimation operations may be performed using only a single target panoramic image, such as analysis of the target panoramic image's visual data via one or more trained neural networks, as described in more detail elsewhere herein.

[0060] Figure 2K Continue to show Figures 2A to 2J, and shows information 255k about additional information that can be generated from one or more images of a room and used in one or more ways in at least some embodiments. In particular, images (e.g., video frames) captured in the living room of house 198 can be analyzed to determine an estimated 3D shape of the living room, such as from a 3D point cloud of features detected in the video frames (e.g., using SLAM and / or SfM and / or MVS techniques, and optionally further based on IMU data captured by one or more image acquisition devices). In this example, information 255k reflects an example portion of such a point cloud of the living room, such as in this example, with Figure 2C Image 250c is similarly oriented to the northwest of the living room (e.g., including the northwest corner 195-1 of the living room and window 196-1). This point cloud can be further analyzed to detect features such as windows, doorways, and other openings between rooms. In this example, region 299 corresponding to window 196-1 and boundary 298 corresponding to the north wall of the living room are identified. It should be understood that in other embodiments, this estimated 3D shape of the living room can be determined by using depth data captured by one or more image acquisition devices in the living room, either in addition to or in lieu of using visual data from one or more images captured by one or more image acquisition devices in the living room. Additionally, it should be understood that various other walls and other features can be similarly identified in the living room and other rooms of house 198.

[0061] Figure 2L Additional information 2551 is shown corresponding to, after determining the final estimated room shapes for the rooms on the illustrated floor of the house 198 (e.g., the 2D room shape 236 for the living room), positioning the estimated room shapes of the rooms relative to each other based, in this example, at least in part, on connecting inter-room corridors between the rooms and matching room shape information for adjacent rooms, which, in at least some embodiments, may be considered constraints regarding the positioning of the rooms and determining an optimal or otherwise preferred solution to those constraints. Figure 2L Examples of such constraints in include making the connecting channel information of adjacent rooms (e.g., in Figures 2E to 2J and / or Figures 2P to 2X) are matched 231 so that the locations of those channels are co-located, and the shapes of adjacent rooms are matched 232 so that those shapes are connected (e.g., as shown for rooms 229d and 229e and rooms 229a and 229b). In other embodiments, various other types of information may be used for room shape locations, either in addition to or in lieu of channel-based constraints and / or room shape-based constraints, such as exact or approximate dimensions of the overall size of the house (e.g., based on available additional metadata about the building, analysis of images from one or more image acquisition locations outside the building, etc.). House exterior information 233 may be further identified and used as a constraint (e.g., based at least in part on automated identification of channels and other features corresponding to the building exterior, such as windows), such as to prevent another room from being placed at a location that has been identified as being outside the building. Figure 2L In the example of , the final estimated room shape used may be a 2D room shape, or instead a 2D version of a 3D final estimated room shape may be generated and used (eg, by taking horizontal slices of the 3D room shape).

[0062] Figure 2M Figure 2-O continues to show Figures 2A to 2L and shows that Figures 2A to 2L and Figure 2P to Figure 2V Mapping information generated by the types of analysis discussed in [1], such as that generated by the MIGM system. In particular, Figure 2M An example 2D floor plan 230m is shown that can be constructed based on the determined location of the final estimated room shape, in this example including indications of walls, doorways, and windows. In some embodiments, such a floor plan may show other information, such as information about other features automatically detected by the analysis operation and / or subsequently added by one or more users. For example, Figure 2NA modified floor plan 230n is shown that includes various types of additional information, such as information that may be automatically identified from analytical operations on visual data from the images and / or from depth data and added to the floor plan 230m, including one or more of the following types of information: room labels (e.g., "Living Room" for a living room), room dimensions, visual indications of furniture or appliances or other fixed features, visual indications of the location of additional types of associated and linked information (e.g., panoramic images and / or stereo images captured at a specified capture location that an end user may select for further display; audio annotations and / or sound recordings that an end user may select for further presentation, etc.), visual indicator lights for doorways and windows, and in other embodiments and scenarios, some or all of this type of information may instead be provided by one or more MIGM system operator users and / or ICA system operator users. Additionally, if assessments and / or other information generated by the BUAM system is available, it may similarly be added to or otherwise associated with floor plans 230m and / or 230n, either in addition to or in place of some or all of the other additional types of information shown for floor plan 230n relative to floor plan 230b. Additionally, when the floor plans 230m and / or 230n are displayed to the end user, one or more user-selectable controls may be added to provide interactive functionality as part of a GUI (graphical user interface) screen 255n, such as indicating the current floor to be displayed, allowing the end user to select a different floor to be displayed, etc., wherein in this example, corresponding example user-selectable controls 228 are added to the GUI. Additionally, in some embodiments, changes to floors or other floors may also be made directly from the displayed floor plan, such as via selection of corresponding connecting pathways (e.g., stairs leading to different floors), and other visual changes may be made directly from the displayed floor plan by selecting corresponding displayed user-selectable controls (e.g., a control selecting a particular image corresponding to a particular location and receiving display of that image, either in place of or in addition to a previous display of the floor plan from which the image was selected). In other embodiments, information for some or all of the different floors may be displayed simultaneously, such as in the form of separate sub-floor plans for the separate floors or, instead, integrating room connection information for all rooms and floors into a single floor plan that is displayed at once. It should be understood that various other types of information may be added in some embodiments, that some of the types of information shown may not be provided in some embodiments, and that visual indications of and user selections of links and associated information may be displayed and selected in other manners in other embodiments.

[0063] Figure 2-0 continues to show Figures 2A to 2N and shows that it is possible to disclose and display (e.g., in a manner similar to Figure 2N2-O ), in some embodiments, additional information may be added to the displayed walls, such as from images taken during video capture (e.g., rendering and illustrating actual paintings, wallpaper, or other surfaces from the house on the rendered model 265), and / or may be used in other ways to add specified colors, textures, or other visual information to the walls and / or other surfaces. Additionally, Figure 2N Some or all of the additional types of information shown in may similarly be added to and shown in the floor plan model 265o.

[0064] Figures 2P to 2X Continue to show Figure 2A To the example of FIG. 2-O, where Figure 2P Further shown is shown Figure 1B Information on part of the living room of house 198 255p. In particular, in Figure 2P In the example of FIG. 2 , an image 250p of the southwest portion of the living room is shown (in a manner similar to Figure 2C 250c ), but with additional information superimposed on the image to illustrate information determined regarding objects and target attributes identified in that portion of the room for further analysis, as well as information regarding the locations of those objects. In particular, in this example, the west window (element 196-2 of image 250c ) is selected as the object of interest in the room, for which a corresponding “west window” label 246p2 has been determined (either automatically or based at least in part on information provided by one or more associated users), and the automatically determined location 199b of the object in the image is shown (in this example, the location is a bounding box for the object). Figure 2P The information 255p further shows a list 248p of objects and properties of objects of interest identified at least in part based on the visual data of the image 250p, indicating that the properties of the object of interest of the west window include its size and information about the view through the window. The image 250p further shows that the door ( Figure 2C1) has been identified as an object of interest, wherein a "front door" label 246p1 is shown (whether automatically or based at least in part on information provided by one or more associated users) and an automatically determined bounding box position 199a. Additionally, information 248p indicates that target attributes of the door include a door handle and door hinges, further visually indicated as 131p on image 250p. Additionally, image 250p also indicates that the sofa ( Figure 2C 191) has been identified as an object of interest, with an automatically determined pixel-level mask position 199c identified for the sofa, but no label is shown in this example. Other objects may be similarly identified, such as one or more ceiling light fixtures as indicated in information 248p, but not shown in example image 250p (e.g., based at least in part on a list of defined types of objects expected or typical for a room of type "living room"). Similarly, other target attributes may be identified, such as the latch hardware for the west window as indicated in information 248p, but not shown in example image 250p (e.g., based at least in part on a list of defined types of target attributes expected or typical for objects of type "window"). Additionally, a "living room" label 246p3 for the room is also determined (whether automatically or based at least in part on information provided by one or more associated users) and shown. In some embodiments, such information 250p and / or information 248p may be displayed, for example, to an associated user in the room (e.g., on the user's mobile computing device or the user's other image acquisition device) as part of providing the user with instructions regarding additional data to be captured, such as to identify specific objects and / or target attributes and their locations. Figure 2Q An alternative image 250q is provided as part of information 255q. In this example, the alternative image is a panoramic image of the 360° visual coverage of the living room. Such a panoramic image can be used in place of or in addition to stereo images such as image 250p to identify object and target attributes and additional related information (e.g., location, label, etc.), as well as for assessing the overall layout of objects in the room and / or the expected flow of traffic through the room. The example panoramic image 250q similarly shows positional bounding boxes 199a and 199b for the front door and west window objects (in this example, rather than the sofa object), as well as an additional positional bounding box 199d for the ceiling light 130b. It should be understood that in other embodiments, various other types of objects and / or target attributes can be identified.

[0065] Figures 2R to 2T Continue to show Figures 2P to 2Q , and further illustrates information about instructions that may be provided to indicate the capture of additional data in the living room and the corresponding additional data captured. In particular, Figure 2RAn image 250r is shown that may be displayed to an associated user in a living room, such as in a manner similar to Figure 2P 250p, but wherein the additional information 249r provides the associated user with instructions for obtaining additional data about the front door object (including target attributes about the hinge and door handle), and the option for the user to receive example and optionally additional instructional information. Figure 2R Not illustrated, but similar instructions may be provided for other objects such as the west window and / or the sofa and / or the ceiling light, such as continuously after instruction 249r has been provided and the corresponding additional data has been obtained, or simultaneously with instruction 249r. Figure 2S An additional image 250s is shown representing additional data captured about the front door in the living room (e.g., in response to instructions provided by the BUAM system), such as having additional details about the door that are not available in the visual data of image 250r. In addition, image 250a is superimposed with examples of additional instructions or other information that may be provided to an associated user (e.g., before or after the user captures the image of the front door shown in image 250a), such as instructions 249s1 to recapture a better-lit image after capturing image 250s based on an automated determination of a corresponding issue with the initial additional image 250a and / or a notification that one or more visible objects before or after capturing image 250s do not appear to be the front door object shown in image 250r (e.g., based on an automated comparison of the visual data in the two images). Figure 2S Further shown is an example 249s2 of additional instructions that may be provided to an associated user regarding additional non-visual data to be captured regarding the front door object (whether before or after capturing image 250s), such as to provide a short text description of the door material and age and / or to record and provide a short video of the door opening including a view through the open door. Figure 2T Further provided are example additional images 250t1 and 250t3 that were captured to provide additional details about the identified target attributes of the front door object, wherein image 250t1 shows additional details about the door handle and image 250t3 shows additional details about one of the hinges. In this example, image 250t1 is further superimposed with example instructions 249t1 to indicate that sufficient details about the door handle were not obtained in image 250t1 (e.g., because the image was not sufficiently focused solely on the door handle) and that a new alternative additional image should be captured, wherein image 250t2 provides an example of such an alternative additional image to be used in place of image 250t1. Image 250t3 further provides an example of additional instructions or other notification information 249t2 to obtain additional data about the visible hinge, in this example, for the associated user to confirm that the visual data in image 250t3 is for the front door object in the living room. It should be understood that in Figures 2P to 2TThe types of instructions and the manner in which they are provided to associated users illustrated in are non-exclusive examples provided for illustrative purposes, and in other embodiments, similar and / or other types of information may be provided in other manners.

[0066] Figure 2U Continue to show Figures 2P to 2T , and provides examples of additional data about a living room that may be obtained based at least in part on analysis of one or more initial room-level images of the living room, such as panoramic image 250q and / or a plurality of stereo images including images 250a-250c and including all or substantially all visual data of the living room. In particular, Figure 2U Information 255u is shown showing alternative examples 237a and 237b of room shapes for a living room (e.g., as may be determined by the MIGM system, as discussed in more detail elsewhere herein), and additional data 236u and 238 of the room shape 237a that may be determined based at least in part on automated operation of the BUAM system and, optionally, additional actions of an associated user. In this example, the shown information 236u provides an example of expected communication traffic information for the living room, such as based at least in part on a determined layout (not shown) of the living room (e.g., using information about furniture and wall openings in the living room). Additionally, the shown information 238 indicates that in this example, the target attribute of the west window may have been assessed as showing a mountain view (e.g., based at least in part on automated determination using visual data visible through the window; at least in part using information from an associated user; at least in part using information from other sources, such as publicly available data, etc.). It should be understood that in Figure 2U These types of additional information described in are non-exclusive examples provided for illustrative purposes, and in other embodiments, similar and / or other types of information may be determined in other manners.

[0067] Figure 2V to Figure 2W Continue to show Figures 2P to 2U and provides examples of additional data about other rooms of a building that can be obtained based at least in part on analysis of one or more initial room-level images of those other rooms. In particular, Figure 2V Information 255v including an image 250v is shown, such as for Figure 1B The bathroom 1 of the example house 198 shown (as in Figure 2N and Figure 2-O). Figure 2PIn the same manner as image 250p of FIG. 1 , image 250v includes an indication 131v of objects and / or target attributes in the bathroom that were identified to capture additional data, including, in this example, a tile floor, a sink countertop, a sink faucet and / or other sink hardware, a bathtub faucet and / or other bathtub hardware, a toilet, etc., however, the location information, labels, and provided instructions are not shown in this example. In a similar manner, Figure 2W Information 255w including an image 250w is shown, such as for Figure 1B The kitchen of the example house 198 shown (as in Figure 2N and Figure 2-O). Figure 2V In the manner of image 250v, image 250w includes an indication 131w of objects and / or target attributes in the kitchen that are identified to capture additional data, in this example, including a refrigerator, a stove on a kitchen island, a sink faucet and / or other sink hardware, a countertop and / or backsplash wall next to the sink, etc. However, in this example, the location information, labels, and provided instructions are not shown. It should be understood that various types of corresponding instructions can be generated and provided to obtain additional data about such identified objects and / or target attributes, and in Figure 2V to Figure 2W These types of additional data are shown in as non-exclusive examples provided for illustrative purposes, such that in other embodiments, similar and / or other types of information may be determined in other manners.

[0068] Figure 2X Continue to show Figures 2P to 2W , and provides examples related to performing an Figure 1B In particular, an example 2.5D or 3D floor plan 265x of a building is shown, which includes information 236x and 247 that can be used as part of a usability assessment of the building. In this example, the floor plan is similar to the one for the living room. Figure 2U In the manner of information 236u, information 236x shows the expected traffic flow pattern through the building. In addition, information 255x further shows information about the determined room types for the various rooms of the building, in this example, graphical symbols are used for different types of corresponding activities for those room types, and wherein in at least some embodiments, those room types and / or corresponding activities can be used as the intended purpose of the corresponding room (e.g., for the bed symbol 247 shown in bedroom 2, sleeping, as shown in FIG. Figure 2N 2-0). It will be appreciated that in some embodiments, information regarding the assessment of specific objects, rooms, and / or buildings may also be overlaid on such floor plans or otherwise provided, and Figure 2XThe additional data types shown in are non-exclusive examples provided for illustrative purposes, such that in other embodiments, similar and / or other types of information may be determined in other manners.

[0069] About Figures 2A to 2X Various details are provided but it is understood that the details provided are non-exclusive examples included for illustrative purposes and that other embodiments may be practiced in other ways without some or all of such details.

[0070] As a non-exclusive example embodiment, the automated operation of the BUAM system may include a subsequent action regarding providing instructions related to capturing additional data for assessing the usability of objects, rooms, and buildings. In this example embodiment, non-exclusive examples of assessing objects of interest and evaluating target attributes may include returning questions such as: Are the kitchen cabinets new? Do they reach up to the ceiling? What are the bathroom fixtures like? What is the condition of the door and window frames? What is the condition of the gutters and downspouts? What is the plumbing under the sink like? What is the hot water tank like? To this end, the BUAM system of the example embodiment may generate and provide instructions and related information such as the following non-exclusive examples: "Take a picture of the kitchen sink," "Zoom in so we can see more detail," "Are you sure that's a sink?", "Thanks for the picture of the tub. Is this from the master bedroom bathroom or the hall bathroom?", "Can you take a close-up of the drainpipe?", etc. As part of doing so, the BUAM system of an example embodiment may perform automated operations to classify or detect common house features (such as sinks, drains, door frames from an image or video), such as building a convolutional neural network model of these, optionally along with a predefined checklist of target attributes (also referred to as "properties" in this example embodiment) for which additional data is to be captured (e.g., for a bathtub, obtain and analyze a close-up image of the drain; for a doorway, obtain and analyze a close-up image of the door jamb, etc.), including verifying that the drain is visible in the corresponding additional image captured and that it is a certain minimum size. As part of doing so, such a BUAM system may provide a GUI (or other user interface) that provides an associated user with a list of identified objects and / or target attributes for which additional data is to be captured, along with corresponding examples of good images of those types.

[0071] The BUAM system of the example embodiment may, for example, implement a workflow having the following steps to assess a room of a house:

[0072] 1) Start with an initial set of images from a house and room or object labels generated by a machine learning model and / or user.

[0073] 2) Given this list of marked rooms and / or objects, generate a list of target attributes to capture or investigate.

[0074] 3) Use the detector model to determine whether the initial image already contains visual data of the target attributes at sufficient image resolution.

[0075] 4) For target attributes that lack such visual data in the initial image, prompt the associated user to capture them, such as in the following manner:

[0076] a. The user is presented with one or more initial images of the room of interest as a "constructive lens".

[0077] b. Optionally, show example images illustrating the details and camera angles to be captured.

[0078] c. Instructing the user to capture images and / or other media (eg, video, 3D model, etc.) having visual data of the indicated target attributes and / or objects.

[0079] d. Analyze captured media through automated onboard processing to:

[0080] i. Verify that the expected data exists at the expected resolution.

[0081] ii. Determine other characteristics to capture.

[0082] e. Optionally, verify that the background in the captured media matches the background in the "constructive shot" or other previously captured images of the room. This verification can be performed, for example, using image information (e.g., by analyzing the background) and / or using telemetry information (e.g., by checking that the camera pose information in the captured media is consistent with the camera pose information in the initial image).

[0083] f. If steps (d) or (e) reveal a problem with the captured media, prompt the user to recapture to correct the problem.

[0084] g. Optionally, prompt the user to enter more data about target attributes and / or objects that cannot be determined visually.

[0085] With respect to step 1 above, the initial images may be panoramic and / or stereo images (e.g., submitted by the seller or agent or photographer during the listing process) and ideally captured separately in each room. They may be annotated with room classification labels upon submission (e.g., a user may label an image as "kitchen," "bedroom," "living room," etc.) and / or may be labeled after submission using a machine learning model for room classification. Additionally, there may be image regions or points where the user has added "point of interest" labels to objects (e.g., "industrial oven" or "new shower"), which may be further used to identify objects of interest and / or associated target attributes. Such operations may be performed, for example, on a mobile computing device used as the image acquisition device and / or on a remote server computing device.

[0086] With respect to step 2 above, the BUAM system may perform a mapping from the tags to target attributes and / or additional data types to be captured. For example, the mapping may indicate information such as, in the following non-exclusive examples: in a kitchen, a close-up of the stove (so the viewer can tell the brand or examine its controls); in a bedroom, a close-up of each sink hardware, bathtub hardware, all sides of the bathtub, etc.; if there is a fireplace, information about whether it burns gas or wood, etc.

[0087] With respect to step 3 above, the BUAM system may use one or more deep learning detector models to detect certain objects and / or target attributes in the image. For example, such detection may include one or more of the following non-exclusive examples: in a kitchen, detecting the stove, sink, and refrigerator; in a bedroom, detecting each sink; in a living room, detecting a fireplace or wood stove, etc. Such a detector model may extract a boundary region from the input image to determine the location of an object (e.g., a <width, height> pixel rectangle whose upper left corner is the sink).<x,y> , auxiliary bounding regions for target attributes such as sink hardware and / or drain pipes, etc.). The BUAM system may use predefined information specifying a minimum expected image size and area in pixels of the corresponding captured additional data for each type of detectable object and target attribute, which the BUAM system will then verify in the visual data of the captured additional images (e.g., to see if they meet the expected size and area).

[0088] With respect to step 4a above, the BUAM system may present an initial image of the bathroom along with a prompt, such as “Please capture a photo / video / 3D model that captures the sink hardware”. With respect to step 4b above, the BUAM system may have a defined library of standard example images for each type of object and target attribute. With respect to step 4c above, the BUAM system may use different types of media in different situations, such as acquiring images of fine detail (and optionally capturing additional data, such as simultaneously capturing a second image using the wide-angle lens of the image acquisition device, providing a narrow / wide perspective pair), short videos assessing functionality (e.g., using a short video of a faucet running at full speed to assess water pressure), assessing 3D models of larger scenes (e.g., using a phone's lidar scanner to capture the perimeter of a house), etc. With respect to step 4d above, the BUAM system may apply models similar to those of step 3 to detect object and target attributes, extract their location areas, and compare them to expected sizes, such operations being performed, for example, on a mobile computing device used as an image acquisition device. Other verification operations may be performed with respect to: image brightness (e.g., if the image is captured in a dark space such as under a cabinet or near a stove / hot water heater), characteristics of the 3D capture (e.g., does the captured 3D model of the house perimeter form a closed loop? - if not, provide instructions to capture the missing area), etc. With respect to step 4e above, the BUAM system may perform verification activities to ensure that the captured image is in the correct room (e.g., checking that the scene context of the captured additional image of the sink is from the correct bathroom, optionally using information from a narrow / wide field of view pair (if available); using image-to-image feature matching to match the visual data in the captured additional image to the visual data of one or more of the initial room-level images; verifying similar colors or textures between the visual data of the captured additional image and one or more of the initial room-level images, etc.), such operations may be performed, for example, on a mobile computing device used as the image acquisition device. With respect to step 4g above, the BUAM system may perform automated operations, such as providing a prompt to enter the year the object (e.g., hot water tank) was last replaced, specifying when the wood floor was last refinished and / or whether it needs refinishing, etc. Additionally, automated operation of the BUAM system may include prioritizing the order in which additional images are captured based on one or more defined criteria, such as capturing visual data and / or other data regarding kitchen appliances before capturing visual data and / or other data regarding kitchen drawer handles (e.g., if kitchen appliance information has greater weight or other impact on usability determinations regarding the kitchen).

[0089] For efficiency purposes, analysis of the visual data of the initial image and / or the captured additional images may include using downsampling of the captured image (to reduce the resolution of the resulting image) if possible, for example, if data is available from a lidar sensor to give 3D geometric information, this may also help in selecting an appropriate amount of downsampling to perform. Additionally, some or all of the operations described above for the example embodiment of the BUAM system may be performed on a mobile computing device used as an image acquisition device and / or may be performed on one or more remote server computing systems (for example, if the operations cannot be performed efficiently or quickly enough on the mobile computing device), in which case there may be a delay between the time the media is captured and the time the relevant feedback is issued, and if so, the feedback in step 4f may be aggregated for all objects and / or target attributes and presented together later.

[0090] Various details have been provided with respect to this example, non-exclusive embodiment, but it is to be understood that the details provided are included for illustrative purposes and other embodiments may be practiced in other ways without some or all of such details.

[0091] Figure 3 30 is a block diagram illustrating an embodiment of one or more server computing systems 300 that execute an embodiment of a BUAM system 340, and one or more server computing systems 380 that execute an embodiment of an ICA system 387 and a MIGM system 388, wherein the one or more server computing systems and the BUAM system may be implemented using a plurality of hardware components that form electronic circuits suitable for and configured to, when operating in conjunction, perform at least some of the techniques described herein. In the illustrated embodiment, each server computing system 300 includes one or more hardware central processing units ("CPUs") or other hardware processors 305, various input / output ("I / O") components 310, storage devices 320, and memory 330, wherein the illustrated I / O components include a display 311, a network connection 312, a computer-readable media drive 313, and other I / O devices 315 (e.g., a keyboard, a mouse or other pointing device, a microphone, a speaker, a GPS receiver, etc.). Each server computing system 380 may include hardware components similar to those of server computing system 300 , including one or more hardware CPU processors 381 , various I / O components 382 , storage devices 385 , and memory 386 , although some details of server 300 are omitted in server 380 for the sake of brevity.

[0092] One or more server computing systems 300 and the executed BUAM system 340 can communicate via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.) with other computing systems and devices, such as: user client computing devices 390 (e.g., for viewing floor plans, associated images, objects and / or room and / or building assessments and / or other related information); one or more ICA and MIGM server computing systems 380; one or more mobile computing devices 360 (e.g., mobile image acquisition devices); optionally one or more camera devices 375; optionally other navigable devices 395 that receive and use floor plans and / or room / building assessment information (e.g., room and building layouts and traffic flow information) and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices); and optionally other computing systems not shown (e.g., for storing and providing additional information related to buildings; for capturing building interior data; for storing and providing information to client computing devices, such as additional supplemental information associated with images and the buildings or other surrounding environments they cover, etc.). In some embodiments, some or all of the one or more camera devices 375 may communicate directly (e.g., to transmit captured target images, receive instructions to initiate capture of target image acquisition or other additional data, etc.) with one or more associated mobile computing devices 360 in their vicinity (e.g., wirelessly and / or via cables or other physical connections, and optionally in a peer-to-peer manner), whether in addition to or in lieu of performing communications via the network 399, and wherein such associated mobile computing devices 360 are capable of providing the captured images and, optionally, other captured data received from the one or more camera devices 375 to other computing systems and devices (e.g., one or more server computing systems 300 and BUAM systems 340, one or more server computing systems 380, etc.) via the network 399.

[0093] In the illustrated embodiment, an embodiment of the BUAM system 340 is executed in memory 330 to perform at least some of the described techniques, such as using one or more processors 305 to execute software instructions of the system 340 in a manner that configures the one or more processors 305 and the one or more computing systems 300 to perform automated operations that implement those described techniques. The illustrated embodiment of the BUAM system may include one or more components (not shown) to each perform a portion of the functionality of the BUAM system, and the memory may further optionally execute one or more other programs 335. As a specific example, in at least some embodiments, a copy of the ICA and / or MIGM system may be executed as one of the other programs 335, such as in place of or in addition to the ICA system 387 and the MIGM system 388 on one or more server computing systems 380. The BUAM system 340 may further store and / or retrieve various types of data on the storage device 320 (e.g., in one or more databases or other data structures) during its operation, such as information 321 about captured room-scale images and information 323 about captured additional images (e.g., with details about objects and / or target attributes of objects); data 324 about the determined room layout and optionally other room-level information (e.g., traffic flow data); data 322 about additional captured data regarding the availability of objects and / or rooms and / or buildings (including assessments of the target attributes of objects); data 325 about generated availability assessments of objects and rooms; data 326 about generated availability assessments of buildings; and data 327 about particular types of rooms and buildings (or factors associated with a room and / or building); data 328 for labeling information in the image (e.g., object label data, room label data, etc.); and optionally various other types of additional information 329 (e.g., information about the user of the client computing device 390 and / or the operator user of the mobile devices 360 and / or 375 interacting with the BUAM system; a list or other predefined information about various types of objects expected in a certain type of room; a list or other predefined information about various types of target attributes expected in a certain type of object and, optionally, a certain type of room; a list or other predefined information about various types of rooms expected in a certain type of building; data about other buildings and their evaluation for comparison, including assessments, etc.).The ICA system 387 and / or the MIGM system 388 may similarly store and / or retrieve various types of data on the storage device 385 (e.g., in one or more databases or other data structures) during their operation and provide some or all of this information to the BUAM system 340 for use (whether in a push or pull manner), such as images 393 (e.g., acquired 360° panoramic images), and optionally other information, such as inter-image directional link information 396 generated by the ICA and / or MIGM systems and used by the MIGM system to generate floor plans; resulting floor plan information and optionally other building mapping information 391 generated by the MIGM system; additional information generated by the MIGM system as part of generating the floor plans (such as determined room shapes 392 and optionally image location information 394); and optionally various types of additional information 397 (e.g., various analytical information related to the presentation or other use of one or more building interiors or other environments captured by the ICA system).

[0094] Some or all of the user client computing devices 390 (e.g., mobile devices), mobile computing devices 360, other navigable devices 395, and other computing systems may similarly include some or all of the same types of components illustrated for the server computing systems 300 and 380. As a non-limiting example, the mobile computing devices 360 are each illustrated as including one or more hardware CPUs 361, I / O components 362, storage devices 365, imaging systems 364, IMU hardware sensors 369, optional depth sensors 363, and memory 367 with a BUAM application 366 and optionally one or both of a browser and one or more other client applications 368 (e.g., ICA system-specific applications) executing within the memory 367, such as to participate in communications with the BUAM system 340, the ICA system 387, and / or other computing systems. Although specific components are not illustrated for the other navigable devices 395 or the client computing system 390, it should be understood that they may include similar and / or additional components.

[0095] It should also be understood that computing systems 300 and 380 and Figure 3The other systems and devices included are merely illustrative and are not intended to limit the scope of the present invention. Systems and / or devices may instead each include multiple interacting computing systems or devices and may be connected to other devices not specifically described, including via Bluetooth communication or other direct communication, through one or more networks (such as the Internet), via the Web, or via one or more dedicated networks (e.g., mobile communication networks). More generally, a device or other computing system may include any combination of hardware that can interact and perform the described types of functions when programmed or otherwise configured with specific software instructions and / or data structures, including but not limited to desktop computers or other computers (e.g., tablet computers, tablet computers, etc.), database servers, network storage devices and other network devices, smartphones and other cellular phones, consumer electronic devices, wearable devices, digital music player devices, handheld gaming devices, PDAs, wireless phones, internet appliances, and various other consumer products that include appropriate communication capabilities. Furthermore, in some embodiments, the functionality provided by the illustrated BUAM system 340 may be distributed among various components, some of the described functionality of the BUAM system 340 may not be provided, and / or other additional functionality may be provided.

[0096] It should also be understood that although various entries are described as being stored in memory or on storage devices when in use, these entries or portions thereof may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments, some or all of the software components and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Thus, in some embodiments, when configured by one or more software programs (e.g., the BUAM system 340 executed on the server computing system 300) and / or data structures, some or all of the described techniques may be performed by hardware components including one or more processors and / or memories and / or storage devices, such as by executing stored contents of software instructions including the one or more software programs and / or by storing such software instructions and / or data structures, and such as by executing algorithms as described in the flowcharts and other disclosures herein. Additionally, in some embodiments, some or all of the systems and / or components may be implemented or provided in other ways, such as by being comprised of one or more devices that are partially or completely implemented in firmware and / or hardware (e.g., rather than as devices that are implemented in whole or in part by software instructions that configure a particular CPU or other processor), including, but not limited to, one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a non-transitory computer-readable storage medium, such as a hard disk or flash drive or other non-volatile storage device, volatile or non-volatile memory (e.g., RAM or flash RAM), network storage, or portable media article (e.g., DVD discs, CD discs, optical discs, flash memory devices, etc.), for reading by an appropriate drive or via an appropriate connection. In some embodiments, the systems, components, and data structures may also be transmitted via generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) over a variety of computer-readable transmission media, including wireless and wired / cable-based media, and may take a variety of forms (e.g., as part of a single or multi-channel analog signal, or as multiple discrete digital packets or frames). In other embodiments, such computer program products may also take other forms. Thus, embodiments of the present disclosure may be practiced using other computer system configurations.

[0097] Figure 4 An example flow chart of an embodiment of an ICA system routine 400 is shown. This routine may be executed by, for example, Figure 1A ICA system 160, Figure 3 The ICA system 387 and / or the ICA system as otherwise described herein may be executed, such as to acquire 360° panoramic images of a target and / or other images within a building or other structure (e.g., for subsequent generation of associated floor plans and / or other mapping information, such as generated by embodiments of the MIGM system routine, wherein the information regarding Figures 5A to 5C An example of such a routine is described; for subsequently evaluating the availability of rooms and buildings, such as by an embodiment of the BUAM system routine, wherein Figures 6A to 6B An example of such a routine is illustrated; for subsequently determining a capture location and, optionally, an capture orientation, etc., of a target image. Although portions of the example routine 400 are discussed with respect to capturing a particular type of image at a particular location, it should be understood that this or similar routines may be used to capture video or other data (e.g., audio, text, etc.) and / or other types of images that are not panoramic, either in lieu of or in addition to such images. Additionally, while the illustrated embodiment captures and uses information from the interior of a target building, it should be understood that other embodiments may implement similar techniques for other types of data, including for non-building structures and / or for information on the exteriors of one or more target buildings of interest. Additionally, some or all of the routines may be executed on a mobile device used by a user to participate in capturing image information and / or related additional data, and / or by a system remote from such mobile device.

[0098] The illustrated embodiment of the routine begins at block 405, where an instruction or information is received. At block 410, the routine determines whether the received instruction or information indicates that data representing a building (e.g., inside a building) should be collected, and if not, continues to block 490. Otherwise, the routine proceeds to block 412 to receive an indication that one or more image acquisition devices are ready to begin an image acquisition process at a first acquisition location. For example, for a mobile computing device acting as an image acquisition device and / or otherwise associated with one or more separate camera devices acting as image acquisition devices, and where the image acquisition devices are carried by an associated user or moved under their own power or by power of one or more other devices that carry or otherwise move the one or more image acquisition devices, the received indication may, for example, come from one of the image acquisition devices, from another powered device that carries or otherwise moves the one or more image acquisition devices, from a user of one or more of the image acquisition devices, etc. After block 412, the routine proceeds to block 415 to perform an acquisition location image acquisition activity to acquire at least one 360° panoramic image (and optionally one or more additional images and / or other additional data, such as from an IMU sensor and / or a depth sensor) for the acquisition location at the target building of interest by at least one image acquisition device, such as to provide at least 360° horizontal coverage around a vertical axis. The routine may also optionally obtain annotations and / or other information about the acquisition location and / or surroundings from the user, such as for later use in presenting information about the acquisition location and / or surroundings.

[0099] After completing block 415, the routine continues to block 417 to determine whether to perform a usability assessment at the current time based on the one or more images captured in block 415, such as with respect to one or more such objects visible from the capture location and / or with respect to one or more visible target attributes of such objects and / or with respect to a room enclosing the capture location, and if so, the routine continues to block 419 to perform automated operations of the building usability assessment manager routine to determine usability assessment information based at least in part on the visual data of the one or more images. Figures 6A to 6B An example of such a building usability assessment manager routine is shown in FIG. After block 419, the routine continues to block 421 to optionally display information regarding usability assessment information determined based at least in part on visual data of the one or more images on one or more of the image acquisition devices (e.g., on a mobile computing device), such as, in some embodiments and scenarios, displaying one or more of the images on the mobile computing device along with overlay information regarding the determined usability assessment information.

[0100] After block 421, or if it is determined in block 417 that usability assessment information is not to be determined for the one or more images captured in block 415 at the current time, the routine continues to block 425 to determine whether there are more capture locations at which images are to be captured, such as based on corresponding provided information (e.g., from one of the image capture devices, from another device that carries or otherwise moves the one or more image capture devices under the power of another device, from a user of one or more of the image capture devices, etc.) and / or meeting specified criteria (e.g., at least two panoramic images to be captured in each of some or all rooms of the building and / or in each of one or more areas outside the building). If so, the routine continues to block 427 to optionally initiate the capture of link information (e.g., visual data, acceleration data from one or more IMU sensors, etc.) during movement of the one or more image capture devices along a path of travel away from the current capture location and toward the next capture location of the building. As described elsewhere herein, the captured linked information may include additional sensor data (e.g., from one or more IMUs or inertial measurement units on one or more of the image capture devices or otherwise carried by a user or other powered device carrying or moving the one or more image capture devices) and / or additional visual information recorded during such movement (e.g., panoramic images, other types of images, panoramic or non-panoramic video, etc.), and in some embodiments, the linked information may be analyzed to determine the changing pose (position and orientation) of the one or more image capture devices during movement, as well as information about the room shape of the enclosed room (or other area) and the path of the image capture devices during movement. Initiation of the capture of such linked information may be performed in response to an explicit received instruction (e.g., from one of the image capture devices, from another device carrying or otherwise moving the one or more image capture devices under the power of another device, from a user of one or more of the image capture devices, etc.) or based on one or more automated analyses of information recorded from the mobile computing device and / or separate camera device. Additionally, in some embodiments, the routine may optionally determine and provide to the user (e.g., by monitoring the movement of one or more of the image capture devices) one or more guidance cues regarding the motion of one or more image capture devices, the quality of the sensor data, and / or visual information captured during movement to the next capture location, including information regarding associated lighting / environmental conditions, the desirability of capturing the next capture location, and any other suitable aspects of capturing link information. Similarly, the routine may optionally obtain (e.g., from the user) annotations and / or other information regarding the path of travel, such as for later use in presenting information regarding the path of travel or resulting inter-panoramic image connection links.In box 429, the routine then determines that the one or more image acquisition devices have arrived at a next acquisition position (e.g., based on an indication from one of the one or more image acquisition devices, from another device that carries or otherwise moves the one or more image acquisition devices under the power of another device, from a user of one or more of the one or more image acquisition devices; based on the forward movement of the one or more image acquisition devices ceasing for at least a predefined amount of time, etc.) to serve as the new current acquisition position, and returns to box 415 to perform image acquisition activities for the new current acquisition position.

[0101] If instead it is determined in block 425 that there are no more acquisition locations at which to acquire image information of the current building or other structure, the routine proceeds to block 430 to optionally analyze the acquisition location information of the building or other structure, such as to identify possible additional coverage (and / or other information) to be acquired within the building or otherwise associated with the building. For example, the ICA system may provide (e.g., to a user) one or more notifications regarding the information acquired during the capture of multiple acquisition locations and, optionally, corresponding linking information, such as if it is determined that the quality of one or more segments of the recorded information is insufficient or unsatisfactory or does not appear to provide complete building coverage, or alternatively may provide corresponding recapture instructions (e.g., to a user, to a device carrying or otherwise moving one or more image acquisition devices, etc.). Additionally, in at least some embodiments, if the captured images (e.g., at least two panoramic images in each room, panoramic images within a maximum specified distance of each other, etc.) do not meet minimum image criteria (e.g., a minimum number and / or type of images), the ICA system may prompt or instruct to acquire additional panoramic images in a similar manner to meet such criteria. After block 430, the routine continues to block 435 to optionally pre-process the acquired 360° target panoramic images, which are then used to generate related mapping information (e.g., to place the images in a spherical format, determine the vanishing lines and points of the images, etc.) In block 480, these images and any associated generated or obtained information are stored for later use.

[0102] If, instead, it is determined in box 410 that the instructions or other information recited in box 405 are not to capture images and other data representing a building, the routine instead continues to box 490 to perform any other indicated operations (such as, any housekeeping tasks) as appropriate, configure parameters for use in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of an image capture device capturing the interior of one or more buildings, an operator user of an ICA system, etc.), obtain and store other information about users of the system, respond to requests for generated and stored information (e.g., requests for use of such information by the MIGM system and / or BUAM system, requests for use of such information by a building map viewer system or display or other presentation by other systems, requests from one or more client devices for display or other presentation of such information, operations for generating and / or training one or more neural networks or another analytical mechanism for use in automated operations of the routine, etc.).

[0103] After block 480 or 490, the routine proceeds to block 495 to determine whether to continue, such as until an explicit termination instruction is received, or instead only if an explicit continuation instruction is received. If it is determined to continue, the routine returns to block 405 to wait for additional instructions or information, and if not, proceeds to step 499 and ends.

[0104] Figures 5A to 5C An example embodiment of a flow chart for a Mapping Information Generation Manager (MIGM) system routine 500 is shown. The routine may be executed by, for example Figure 1A MIGM system 160, Figure 3 The MIGM system 388 and / or execution of the MIGM system as described elsewhere herein may be performed, such as to determine a room shape of a room (or other defined area) by analyzing and combining information from one or more images captured in the room, to generate a floor plan of a building or other defined area based at least in part on the one or more images of the area and optionally additional data captured by the mobile computing device, and / or to generate other mapping information of a building or other defined area based at least in part on the one or more images of the area and optionally additional data captured by the mobile computing device. Figures 5A to 5CIn the example of , the determined room shape of a room may be a 3D fully enclosed combination of planar surfaces representing the walls, ceiling, and floor of the room, or may have other forms (e.g., based at least in part on a 3D point cloud), and the generated mapping information of a building (e.g., a house) includes a 2D floor plan and / or a 3D computer model floor plan and / or a set of linked images, where the links indicate relative orientations between various pairs of images, but in other embodiments, other types of room shapes and / or mapping information may be generated and used in other ways, including other types of structures and defined areas, as discussed elsewhere herein.

[0105] The illustrated embodiment of the routine begins at block 505, where information or instructions are received. The routine continues to block 510 to determine whether image information is currently available for analysis for one or more rooms (e.g., for some or all of the indicated building), or whether such image information is currently being acquired. If it is determined in block 510 that some or all of the image information is currently being acquired, the routine continues to block 512 to acquire such information, optionally while one or more image acquisition devices are moved through one or more rooms of the building (e.g., carried by an associated user or moved under their own power or by power of one or more other devices that carry or otherwise move the one or more image acquisition devices) and acquire panoramic or other images at one or more acquisition locations in the one or more rooms (e.g., multiple acquisition locations in each room of the building), optionally along with metadata information regarding the acquisition and / or interconnection information related to the movement between the acquisition locations, as discussed in more detail elsewhere herein. Figure 4 An example embodiment of an ICA system routine for performing such image acquisition is provided. If instead it is determined at block 510 that an image is not currently being acquired, the routine instead continues to block 515 to obtain an existing panoramic or other image from one or more acquisition locations in one or more rooms (e.g., multiple acquisition locations in each room of a building), optionally along with metadata information regarding the acquisition and / or interconnection information related to movement between acquisition locations, such as may in some cases have been supplied in block 505 and / or previously obtained by the ICA system along with corresponding instructions.

[0106] After either block 512 or 515, the routine continues to block 520 where it determines whether a set of linked target panoramic images (or other images) of a building or other set of rooms is to be generated, and if so, continues to block 525. In block 525, the routine selects pairs of at least some of the images (e.g., based on the images of a pair having overlapping visual content and / or based on link information connecting the images of a pair), and for each pair, determines a relative orientation between the images of the pair (whether movement from the capture location of one image of the pair directly to the capture location of the other image of the pair, or instead movement via one or more other intermediate capture locations of other images between those starting and ending capture locations) based on the shared visual content and / or other captured link interconnection information (e.g., movement information) related to the images of the pair. In block 525, the routine further uses at least the relative orientation information for the image pair to determine the global relative positions of some or all of the images to each other in a common coordinate system, such as to create a virtual tour by which an end user can move from any one of the images to one or more other images linked to the starting image (e.g., via selection of a user-selectable control displayed for each such other linked image (such as superimposed on the displayed image)), and similarly from the next image to one or more additional images linked to the next image, etc. Additional details regarding creating such a set of linked images are included elsewhere herein.

[0107] Following box 525, or if it is instead determined in box 520 that the instruction or other information received in box 505 is not to determine a set of linked images, the routine continues to box 530 to determine whether the instruction received in box 505 indicates determining the shape of one or more rooms from previously or currently captured images in the rooms (e.g., from one or more panoramic images captured in each of the rooms), and if so, continues to box 550, otherwise continues to box 590.

[0108] In block 550, the routine proceeds to selecting the next room (starting with the first one) for which one or more panoramic images and / or other images captured in the room are available, and determining a 2D and / or 3D shape of the room based at least in part on visual data of the one or more images captured in the room and / or additional data captured in the room, including optionally obtaining additional metadata for each image (e.g., acquisition height information of a camera device or other image capture device used to capture the image). Determining the room shape of the room may include analyzing visual content of the one or more images captured in the room by the one or more image capture devices and / or analyzing additional non-visual data captured in the room (e.g., by the one or more image capture devices), including determining initial estimated acquisition pose information (e.g., acquisition position and optionally acquisition orientation) for each of the images. Analysis of the various data collected in the room may also include identifying features of wall structural elements of the room (e.g., windows, doorways, and stairs, as well as other inter-room wall openings and connecting passages, wall boundaries between a wall and another wall and / or a receiving piece and / or a floor, etc.) and determining the locations of those identified features within the determined room shape of the room, optionally by generating a 3D point cloud of some or all of the walls and optionally the ceiling and / or floor of the room (e.g., by analyzing at least visual data of images collected in the room and optionally additional data captured by one or more of the image collection devices, such as using one or more of SfM or SLAM or MVS analysis), and / or by determining planar surfaces corresponding to some or all of the walls and optionally the floor and / or ceiling (e.g., by determining normals / orthogonal directions to the planes of the identified features and combining such information to determine wall position hypotheses and optionally clustering multiple wall position hypotheses for a given wall to achieve a final determination of the position of that wall). Additional details regarding determining the room shape and identifying additional information for the room, including initial estimated collection pose information for images collected in the room, are included elsewhere herein.

[0109] After block 550, the routine continues to block 567 to determine whether there are more rooms for which one or more captured images are available, and if so, returns to block 550 to determine the room shape of the next such room. Otherwise, the routine continues to block 537 to determine whether a floor plan of the indicated building is generated (e.g., based at least in part on the room shape determined in block 550), and if not, continues to block 590. Otherwise, the routine continues to block 537, where the routine optionally obtains additional information about the building, such as from activities performed during the acquisition and optional analysis of the images, and / or from one or more external sources (e.g., online databases, information provided by one or more end users, etc.), such additional information may include, for example, the exterior dimensions and / or shape of the building, additional images and / or annotation information acquired corresponding to specific locations within the building (optionally for locations different from the acquisition location of the acquired panoramic or other image), additional images and / or annotation information acquired corresponding to specific locations outside the building (e.g., around the building and / or for other structures on the same property), etc.

[0110] After block 537, the routine proceeds to block 575 to retrieve room shapes (e.g., the room shapes generated in block 550) or otherwise obtain room shapes for the rooms of the building (e.g., based on human-provided input), whether 2D or 3D room shapes. The routine then proceeds to block 577, where the routine uses the determined room shapes to create an initial 2D floor plan, such as by connecting the inter-room corridors in their respective rooms, by optionally positioning the room shapes around the determined acquisition locations of the images (e.g., if the acquisition locations are interconnected), and by optionally applying one or more constraints or optimizations. Such a floor plan may include, for example, relative position and shape information for various rooms without providing any actual size information for individual rooms or the building as a whole, and may further include multiple linked or associated sub-maps of the building (e.g., to reflect different floors, levels, sections, etc.). The routine further associates the locations of doors, wall openings, and other identified wall elements with the floor plan. After block 577, the routine optionally performs one or more steps 580 to 583 to determine and associate additional information with the floor plan. In block 580, the routine optionally estimates the dimensions of some or all of the rooms, such as based on analysis of the image and / or its acquisition metadata or based on overall dimensional information obtained for the exterior of the building, and associates the estimated dimensions with the floor plan. After block 580, the routine continues to block 583 to optionally associate further information with the floor plan (e.g., with a specific room or other location within the building), such as additional images and / or annotation information for the specified location. In block 585, if the room shape from block 575 is not a 3D room shape, the routine further estimates the heights of some or all of the walls in the room, such as based on analysis of the image and optionally the sizes of known objects in the image, and height information about the camera at the time the image was acquired, and uses this height information to generate a 3D room shape for the room. The routine further uses the 3D room shape (whether from block 575 or block 585) to generate a 3D computer model floor plan of the building, wherein the 2D and 3D floor plans are associated with each other. In block 485, the routine then optionally calls the MIGM system to perform further analysis using the information obtained and / or generated in routine 400, such as to generate a partial or full floor plan of the building and / or generate other mapping related information, and wherein Figures 5A to 5C An example of a routine for such a MIGM system is provided by routine 500. It will be appreciated that if sufficiently detailed dimensional information is available, architectural drawings, blueprints, etc. may be generated from the floor plans.

[0111] After box 585, the routine continues to box 588 to store the determined one or more room shapes and / or the generated mapping information and / or other generated information, and optionally further use some or all of the determined and generated information, such as to provide the generated 2D floor plan and / or 3D computer model floor plan for display on one or more client devices and / or to one or more other devices for automated navigation of those devices and / or associated vehicles or other entities, to similarly provide and use information about the determined room shape and / or a set of linked panoramic images and / or about additional information determined based on the contents of the room and / or pathways between rooms, to provide information as a response to another routine that calls routine 500, and the like.

[0112] If it is determined in block 530 that the information or instructions received in block 505 are not to determine the shape of one or more rooms, or if it is determined in block 535 that the information or instructions received in block 505 are not to generate a floor plan of the indicated building, the routine continues to block 590 to perform one or more other indicated operations, as appropriate. Such other operations may include, for example, receiving and responding to requests for previously generated floor plans and / or previously determined room shapes and / or previously determined sets of linked images and / or other generated information (e.g., requests for such information to be used for automated navigation by one or more other devices, requests for such information to be used by the BUAM system, requests for such information to be displayed or otherwise presented by a building map viewer system or other system, requests from one or more client devices for display or other presentation of such information, operations for generating and / or training one or more neural networks or other analysis mechanisms for automated operation of the routine, etc.), obtaining and storing information about the building for later operation (e.g., information about the size, number, or type of rooms, total floor area, other adjacent or nearby buildings, adjacent or nearby vegetation, exterior imagery, etc.), etc.

[0113] After block 588 or 590, the routine continues to block 595 to determine whether to continue, such as until an explicit termination instruction is received, or instead to continue only if an explicit continue instruction is received. If it is determined to continue, the routine returns to block 505 to wait for and receive additional instructions or information, otherwise, it continues to block 599 and ends.

[0114] Although not relative to Figures 5A to 5CThe automated operations illustrated in the example embodiments of the present invention are described herein, but in some embodiments, a human user may further help facilitate some operations of the MIGM system, such as an operator-user and / or end-user of the MIGM system providing one or more types of input that are further used for subsequent automated operations. As non-exclusive examples, such a human user may provide one or more types of input, as follows: providing input to help link a set of images, such as providing input in box 525 that is used as part of the automated operations of that box (e.g., to specify or adjust the initial automatically determined orientation between one or more pairs of images, to specify or adjust the initial automatically determined final global position of some or all images relative to each other, etc.); providing input in box 537 that is used as part of the subsequent automated operations, such as one or more of the information of the shown type about the building; providing input with respect to box 550 that is used as part of the subsequent automated operations, such as to specify or adjust the automatically determined orientation of one or more of the images; , to specify or adjust the initial automatically determined position of a room shape within a generated floor plan and / or to specify or adjust the initial automatically determined room shape itself within such a floor plan; to provide input with respect to one or more of blocks 580, 583, and 585, to be used as part of subsequent operations, such as to specify or adjust one or more types of initially automatically determined information discussed with respect to those blocks, etc. Additional details regarding embodiments in which one or more human users provide input that is further used for additional automated operations of the MIGM system are included elsewhere herein.

[0115] Figures 6A to 6B An example flow chart for a Building Usability Assessment Manager (BUAM) system routine 600 is shown. The routine may be executed by, for example Figure 1A BUAM system 140 and / or BUAM application 156, Figure 3 The BUAM system 340 and / or BUAM application 366 and / or as described in relation to Figures 2P to 2X and execution of the BUAM system described elsewhere herein, such as to perform automated operations related to analyzing visual data from images captured in rooms of a building and optionally appending the captured data to assess the rooms of the building and objects in the rooms (e.g., based at least in part on automated assessment of one or more target attributes of each object) as well as the room layout and other usability information of the building itself. Figures 6A to 6BIn the examples shown, a specific type of captured data is used and analysis is performed in a particular manner, but in other embodiments, other types of information may be obtained, analyzed, and used in other ways. Additionally, while the illustrated embodiment captures and uses information from the interior of a target building, it should be understood that other embodiments may perform similar techniques for other types of data, including information for non-building structures and / or for the exterior of one or more target buildings of interest. Additionally, some or all of the routines may be performed on a mobile device (e.g., a mobile computing device or other image capture device) used by a user to participate in the capture of image information and / or related additional data, and / or by a system remote from such a mobile device.

[0116] The illustrated embodiment of the routine begins at block 605, where information or instructions are received. The routine continues to block 610 to determine whether the instructions or other information indicate that the availability of one or more indicated rooms should be evaluated (e.g., for some or all rooms of a building). If not, the routine continues to block 690, and otherwise continues to block 615, where the routine selects the next indicated room (starting with the first one) and obtains one or more initial images of the room whose visual data includes most or all of the room (e.g., images previously acquired by the ICA system; images acquired simultaneously in the room by one or more image acquisition devices, such as in an automated manner and / or using one or more associated users participating in image acquisition, and optionally in response to corresponding instructions initiated by the routine and provided to the image acquisition devices and / or associated users, etc.). The one or more initial images are then analyzed to identify one or more objects of interest in the room for which more data is to be collected (if the one or more initial images do not include sufficient detail about the objects in their visual data, if types of data other than visual data are not available in the information captured and obtained about the one or more images, etc.), and optionally to determine additional information about the room, such as to assess a label or other type or category information for the room, to assess the shape of the room and / or the layout of items in the room, to assess expected traffic patterns for the room (e.g., based at least in part on the layout and / or shape) and / or actual traffic patterns for the room (e.g., if there are sufficient images to show people moving through the room), etc. In some embodiments, while identifying objects based at least in part on analysis of visual data (e.g., in a joint or other related manner), additional information about some or all of the objects is additionally determined, such as object locations, object tags or other type or category information for the objects, etc. Alternatively, in some embodiments and scenarios, at least some of this information (e.g., one or more tags or other type or category information) may be obtained for the room and / or one or more objects in the room in other ways, such as from previously generated information (e.g., generated by an ICA system) and / or from concurrently generated information (e.g., based at least in part on information from one or more users in the room participating in concurrent image acquisition, etc.). Additional details regarding determining information about a room based at least in part on visual data of one or more initial images captured in the room are included elsewhere herein.

[0117] After box 615, the routine continues to box 620 where the routine selects the next object identified in box 615 (starting with the first one) in the current room and determines additional information about the object, such as by analyzing the visual data in the one or more initial images and optionally using other data to determine the type and / or category of the object (if not determined in box 615), to determine one or more target attributes of interest about the object for which additional data is to be collected (e.g., based at least in part on the type or category of the object, such as from a predefined list of some or all such target attributes for objects of that type or category), to determine the location of the object (if not determined in box 615), etc. As part of doing so, the routine may further analyze the visual data of the one or more initial images to verify whether the visual data includes sufficient detail about each of the target attributes and, if sufficient detail is already available, not include the target attributes in the additional data to be captured (and other non-visual types of additional data are not identified as to be captured). The routine further determines one or more types of additional data (whether one or more additional images or one or more other types of additional data) to be collected for each of the target attributes that is not already available, generates corresponding instructions to direct the automated capture of the additional data and / or direct an associated user to participate in the capture of the additional data, and provides the instructions to the one or more image acquisition devices and / or users, optionally along with examples (or, if desired, access to such examples, such as via a user-selectable link). The routine then further collects the additional data about the object and its target attributes from the one or more image acquisition devices and / or associated users. Additional details regarding determining information about the object based at least in part on visual data of one or more initial images captured of a room in which the object is located are included elsewhere herein.

[0118] After box 620, the routine continues to box 625, in which the routine optionally evaluates the additional data captured in box 620 to identify possible problems (e.g., incorrect object and / or target attributes visible in the additional images and / or depicted in other data, insufficient detail in the visual data or other data of the additional images to enable assessment of target attributes and / or evaluation of the object, other types of image problems or other types of data problems, etc.), and if so, initiates corrective action (e.g., providing other instructions to one or more image acquisition devices and / or associated users to capture additional images and / or other data that correct the problems), including obtaining any such corrected additional images and / or other data to supplement or replace the initial additional images and / or other initial additional data having the identified possible problems. Additionally, while the acquisition of the initial image, additional images, and optionally other additional data is illustrated in blocks 615 through 625 as occurring after the instructions are provided and before proceeding to the next block of the routine, it should be understood that the acquisition of such images and / or other data may occur substantially immediately (e.g., concurrently with the instructions, such as in an interactive manner) and / or asynchronously (e.g., a significant amount of time after the instructions are provided, such as minutes, hours, days, etc.), and that the routine may perform other operations (e.g., for other rooms and / or other buildings) while awaiting the images and optionally other additional data.

[0119] After block 625, the routine continues to block 630 where it determines whether there are any identified objects in the current room, and if so, returns to block 620 to select the next such object. Otherwise, the routine continues to block 635 where, for each identified target attribute of each of the identified objects in the current room, the routine analyzes the captured additional data available about the target attribute to evaluate the target attribute of the object relative to one or more defined factors or other defined attribute criteria for that type of target attribute and / or object. In at least some embodiments, the evaluation of the target attribute is performed to estimate the current contribution of the target attribute to the usability assessment of the object, such as the object's contribution to the overall usability of the room for its intended purpose. After block 635, the routine continues to block 640 where, for each identified object in the current room, the captured additional data and other information about the room are analyzed to assess the object's usability relative to one or more defined object criteria for objects of that type and / or enclosing the room, such as to assess the object's contribution to the room's overall usability for the room's intended purpose, including using information from the assessment of one or more target attributes of the object (e.g., combining assessment information from multiple target attributes) and optionally further using additional information about the object. In block 645, the routine then assesses the room's overall usability for the intended purpose, such as relative to one or more defined room criteria for rooms of that type, including using information from the assessment of one or more identified objects in the room (e.g., combining assessment information from multiple identified objects), and optionally further using additional information about the room (e.g., an assessed room layout, estimated traffic flow patterns for the room, etc.).

[0120] Following box 645, the routine continues to box 650 where the routine determines whether there are additional indicated rooms to be evaluated and, if so, returns to box 615 to select the next such room. In at least some embodiments and scenarios, the determination of whether additional rooms exist may be made dynamically based at least in part on information from one or more image capture devices and / or associated users in the room, such as if, as part of the next iteration of the operations of box 615, one or more image capture devices and / or associated users move to the next room in the building and interactively advance to obtain one or more initial images of that next room (or instead indicate that the last room of the building has been captured so that there are no more rooms). Otherwise, the routine continues to block 685 where, if multiple rooms of the building have been evaluated, the routine optionally evaluates the overall usability of the building relative to one or more defined building evaluation criteria (such as relative to the intended purpose of the building). As part of doing so, the routine may use information from the evaluations of the rooms in the building (e.g., to combine evaluation information from multiple rooms) and further optionally use additional information about the building (e.g., the evaluated building layout, estimated traffic flow patterns for the building, etc.). Additional details regarding the assessments and evaluations performed relative to blocks 635 through 645 and 685 are included elsewhere herein.

[0121] After block 685, the routine continues to block 688 where the routine stores the information determined and generated in blocks 615 through 685 and optionally displays some or all of the determined and / or evaluated information and / or optionally provides some or all of the determined and / or evaluated information for further use (e.g., for automated operation of a building by one or more devices; in response to another routine invoking routine 600, such as with respect to Figure 4 419, etc.).

[0122] If it is instead determined in block 610 that the instruction or information received in block 605 was not to evaluate the availability of one or more indicated rooms, the routine instead continues to block 690 where it performs one or more other indicated operations as appropriate. Such other operations may include, for example, one or more of the following: receiving and responding to requests for evaluations and / or other generated information of previously generated rooms, buildings, and / or objects (e.g., requests for such information for use by one or more other devices for automated navigation, requests for such information for display or other presentation by a building map viewer system or other system, requests from one or more client devices for display or other presentation of such information, etc.); operations for generating and / or training one or more neural networks or another analysis mechanism for routine automated operations; obtaining and storing information about buildings, rooms, objects, and / or object target attributes for later operations (e.g., information about expected or typical target attributes for particular objects or object types, information about expected or typical objects for particular rooms or room types, information about expected or typical rooms for particular buildings or building types, information about one or more types of defined criteria for automated analysis, information about factors for assessing particular target attributes or target attribute types or types of associated objects, etc.); information about the intended purposes of particular rooms and / or room types and / or buildings and / or building types, etc.

[0123] After blocks 688 or 690, the routine continues to block 695, where the routine determines whether to continue, such as until an explicit termination indication is received, or instead not to continue unless an explicit continue indication is received. If it is determined to continue, the routine returns to block 605; otherwise, it continues to block 699 and ends. It should be understood that while the example embodiment of the routine 600 receives information regarding the evaluation of one or more rooms and, optionally, a multi-room building and proceeds to perform such activities, other embodiments of the routine may analyze other levels of information, such as instead assessing one or more indicated target attributes (e.g., without further evaluating one or more objects corresponding to the target attributes), evaluating one or more indicated objects (e.g., without further evaluating one or more rooms in which the one or more objects are located), etc.

[0124] Figure 7 An example embodiment of a flow chart for a building map viewer system routine 700 is shown. The routine may be executed, for example, Figure 1A The map viewer client computing device 175 and one or more software systems thereof (not shown), Figure 3The client computing device 390 and / or mobile computing device 360, and / or a mapping information viewer or presentation system as described elsewhere herein, performs, such as: receiving and displaying a determined room shape and / or other mapping information (e.g., a 2D or 3D floor plan) defining an area, the mapping information optionally including one or more determined image acquisition locations and / or one or more generated visual indications of usability assessments (e.g., associated with a particular location in the mapping information); and optionally displaying additional information associated with the particular location (e.g., an image, optionally with usability assessment information superimposed or otherwise associated with one or more objects and / or rooms visible in the image) in the mapping information. Figure 7 In the example shown, the mapping information presented is for the interior of a building (such as a house), but in other embodiments, other types of mapping information may be presented for other types of buildings or environments and used in other ways, as discussed elsewhere in this document.

[0125] The illustrated embodiment of the routine begins at block 705, where an instruction or information is received. At block 710, the routine determines whether the received instruction or information indicates displaying or otherwise presenting information representing the interior of a building, and if not, proceeds to block 790. Otherwise, the routine proceeds to block 712 to retrieve one or more room shapes or floor plans of the building or other generated mapping information of the building, and optionally, an indication of associated linked information about surrounding locations within the building and / or outside the building, and optionally, an indication of information (e.g., usability assessment information) to overlay on or otherwise associate with the mapping information, and selects an initial view of the retrieved information (e.g., a view of the floor plan, a specific room shape, etc.). In block 715, the routine then displays or otherwise presents the current view of the retrieved information and awaits user selection in block 717. After the user selection in block 717, if it is determined in block 720 that the user selection corresponds to adjusting the current view of the current location (e.g., changing one or more aspects of the current view, adding usability assessment information superimposed on or otherwise associated with the current view, etc.), the routine continues to block 722 to update the current view in accordance with the user selection, and then returns to block 715 to update the displayed or otherwise presented information accordingly. The user selection and corresponding update of the current view may include, for example, displaying or otherwise presenting a piece of associated link information selected by the user (e.g., a particular image associated with the displayed visual indication of the determined acquisition location, such as superimposing the associated link information on at least some of the previously displayed information) and / or changing the display manner of the current view (e.g., zooming in or out; rotating the information as appropriate; selecting a new portion of the floor plan to be displayed or otherwise presented, such as where some or all of the new portion was not previously visible, or where instead the new portion is a subset of the previously visible information, etc.).

[0126] If it is instead determined in box 710 that the instructions or other information received in box 705 will not present information representing the interior of a building, the routine instead continues to box 790 to perform other indicated operations (such as, any housekeeping tasks) as appropriate, configure parameters for use in various operations of the system (e.g., based at least in part on information specified by a user of the system, such as a user of a mobile device capturing the interior of one or more buildings, an operator user of the MIGM system, etc., including for personalizing the display of information for a particular user according to his / her preferences), obtain and store other information about users of the system, respond to requests for generated and stored information, and the like.

[0127] After block 790, or if it is determined in block 720 that the user selection does not correspond to the current building area, the routine proceeds to block 795 to determine whether to continue, such as until an explicit termination instruction is received, or instead only if an explicit continue instruction is received. If it is determined to continue (including if the user makes a selection in block 717 regarding a new location to be presented), the routine returns to block 705 to wait for additional instructions or information (or if the user makes a selection in block 717 regarding a new location to be presented, then proceed directly to block 712), and if not, proceeds to step 799 and ends.

[0128] Non-exclusive example embodiments described herein are further described in the following clauses.

[0129] A01. A computer-implemented method for one or more computing systems to perform automated operations, the method comprising:

[0130] obtaining, by one or more computing systems, one or more images captured in a room of a house;

[0131] analyzing, by the one or more computing systems, visual data of the one or more images to identify a plurality of objects installed in the room and determining, for each of the plurality of objects, at least one type of the object;

[0132] determining, by the one or more computing systems and for each of the plurality of objects, one or more target attributes of the object based at least in part on the type of the object;

[0133] obtaining, by the one or more computing systems, additional images captured in the room to each provide additional visual data having additional details regarding at least one target attribute of at least one of the plurality of objects;

[0134] analyzing, by the one or more computing systems and via use of at least one trained neural network, the additional visual data of the additional images to evaluate the plurality of objects, including determining, for each of the plurality of objects, a current contribution of the object to the room for the indicated purpose based at least in part on assessing the one or more target attributes of the object using information determined from the additional visual data;

[0135] determining, by the one or more computing systems and based at least in part on combining the determined current contributions of the plurality of objects, an assessment of the availability of the room for the indicated purpose; and

[0136] Information regarding the determined assessment of the availability of the room is provided by the one or more computing systems.

[0137] A02. A computer-implemented method for one or more computing systems to perform automated operations, the method comprising:

[0138] obtaining, by one or more computing systems, one or more panoramic images captured in a room of a house and having visual data, the one or more panoramic images combining to provide 360-degree horizontal visual coverage of the room;

[0139] analyzing, by the one or more computing systems and via use of at least a first trained neural network, the visual data of the one or more panoramic images to identify a plurality of objects installed in the room and assess a layout of the room relative to a designated purpose of the room;

[0140] determining, by the one or more computing systems and for each of the plurality of objects, one or more target attributes of the objects for which additional data is to be captured based at least in part on a lack of detail in the visual data satisfying a defined threshold for the one or more target attributes of the objects;

[0141] providing, by the one or more computing systems, instructions to capture the additional data regarding the one or more target attributes of each of the plurality of objects, wherein capturing the additional data comprises obtaining additional stereoscopic images of one or more specified types of the plurality of objects;

[0142] analyzing, by the one or more computing systems and via use of at least a second trained neural network, the visual data of the additional stereoscopic images to verify, for each of the plurality of objects, that the additional data regarding the one or more target properties of the object has been captured, and to generate an assessment of the object's current contribution to the usability of the room for the purpose of indicating based at least in part on the one or more target properties of the object;

[0143] determining, by the one or more computing systems and based at least in part on combining information regarding the assessed layout of the room and the assessed current contributions of the plurality of objects, an assessment of the current availability of the room for the indicated purpose; and

[0144] Information regarding the determined assessment of the current availability of the room and additional visual information regarding the room is displayed by the one or more computing systems.

[0145] A03. A computer-implemented method for one or more computing systems to perform automated operations, the method comprising:

[0146] obtaining, by the one or more computing systems, a plurality of images captured in a room of a building;

[0147] analyzing, by the one or more computing systems and via use of at least one trained neural network, visual data of the plurality of images to identify a plurality of objects in the room;

[0148] determining, by the one or more computing systems and for each of the plurality of objects, one or more target attributes of the object, and determining a contribution of the object to the usability of the room for purposes of indication based at least in part on assessing the one or more target attributes of the object using the visual data of the plurality of images;

[0149] determining, by the one or more computing systems and based at least in part on combining the determined contributions of the plurality of objects, an assessment of the availability of the room for the indicated purpose; and

[0150] Information regarding the determined assessment of the availability of the room is provided by the one or more computing systems.

[0151] A04. A computer-implemented method for one or more computing systems to perform automated operations, the method comprising:

[0152] obtaining one or more images captured in a room of a building;

[0153] analyzing visual data of the one or more images to identify one or more objects in the room and determine a type of each of the one or more objects;

[0154] identifying one or more target attributes for each of the one or more objects based at least in part on the determined type of the object, and obtaining additional visual data of the room having additional details regarding at least one target attribute for at least one of the one or more objects;

[0155] determining, for each of the one or more objects, a contribution of the object to the usability of the room based at least in part on assessing the one or more target attributes of the object based at least in part on the additional data;

[0156] further analyzing at least one of the visual data or the additional visual data to assess at least one of a shape of the room and a layout of items in the room;

[0157] determining the assessment of the usability of the room based at least in part on combining information regarding the determined contributions of the one or more objects with information regarding the assessed shape or layout of the room; and

[0158] Information regarding the determined assessment of the availability of the room is provided.

[0159] A05. A computer-implemented method according to any one of clauses A01 to A04, wherein the analysis of the visual data of the one or more panoramic images by the one or more computing systems further includes: determining the position of each of the plurality of objects in the room and determining the type of the room, wherein providing the instructions to capture the additional data about the one or more target attributes of each of the plurality of objects includes providing information about the determined position of the object and about the additional stereoscopic images of the one or more specified types to be obtained for the object, and the method further includes: determining, by the one or more computing systems, the purpose of the indication of the room based at least in part on the determined type of the room.

[0160] A06. The computer-implemented method of any of clauses A01 to A05, wherein the house comprises a plurality of rooms, and wherein the method further comprises:

[0161] obtaining and analyzing visual data, and determining said one or more target attributes, and providing said instructions, and analyzing said additional visual data, and determining said assessment of said current usability of said room, performed by said one or more computing systems and for each of said plurality of rooms; and

[0162] determining, by the one or more computing systems, an assessment of the overall usability of the home based at least in part on combining the assessment of the layout of the plurality of rooms of the home and information regarding the determined assessment of the current usability of each of the plurality of rooms, and

[0163] Wherein displaying the information by the one or more computing systems further comprises displaying information regarding the determined assessment of the overall usability of the premises superimposed on a displayed floor plan of the premises.

[0164] A07. The computer-implemented method of any of clauses A01 to A06, wherein analyzing the visual data of the one or more images comprises:

[0165] analyzing, by the one or more computing systems, the visual data of the one or more images to determine an amount of information in the visual data regarding the one or more target attributes of each of the plurality of objects; and

[0166] determining, by the one or more computing systems, to capture the additional image based at least in part on the determined amount of information in the visual data for each of the plurality of objects being insufficient to satisfy a defined detail threshold for at least one target attribute of the object, and

[0167] Wherein obtaining the additional image is performed based at least in part on determining to capture the additional image.

[0168] A08. The computer-implemented method of clause A07, wherein analyzing the visual data of the one or more images to identify the plurality of objects further comprises identifying one or more additional objects in the room separate from the plurality of objects, and wherein the method further comprises:

[0169] analyzing, by the one or more computing systems, the visual data of the one or more images to determine a further amount of information in the visual data regarding one or more additional target attributes of each of the one or more additional objects; and

[0170] A determination by the one or more computing systems not to capture further additional images is based at least in part on the determined further amount of information in the visual data for each of the one or more additional objects being sufficient to satisfy a defined detail threshold for the one or more additional target attributes of the additional object.

[0171] A09. The computer-implemented method of any of clauses A07 to A08, further comprising:

[0172] analyzing, by the one or more computing systems, the additional visual data of the additional images to determine a further amount of information in the additional visual data regarding the one or more target attributes of each of the plurality of objects; and

[0173] determining, by the one or more computing systems, that the further amount of information in the additional visual data is sufficient to satisfy a defined detail threshold for the one or more target attributes of each of the plurality of objects, and

[0174] Wherein determining the current contribution of each of the plurality of objects is performed at least in part based on determining that the further amount of information in the additional visual data is sufficient to satisfy a defined detail threshold of the one or more target attributes of each of the plurality of objects.

[0175] A10. The computer-implemented method of any of clauses A07 to A09, further comprising:

[0176] analyzing, by the one or more computing systems, the additional visual data of the additional image to determine a further amount of information in the additional visual data regarding the one or more target attributes of each of the plurality of objects;

[0177] determining, by the one or more computing systems and for one of the plurality of objects, that the further amount of information in the additional visual data is insufficient to satisfy a defined detail threshold of at least one target attribute of the one object; and

[0178] Initiate, by the one or more computing systems and based on determining that the further amount of information in the additional visual data is insufficient to satisfy a defined detail threshold for the at least one target attribute of the one object and before determining the current contribution of the one object, capture one or more further images of the one object to provide further visual data sufficient to satisfy the defined detail threshold for the one or more target attributes of the one object.

[0179] A11. A computer-implemented method according to any one of clauses A01 to A10, further comprising comparing, by the one or more computing devices, the additional visual data of the additional images with the visual data of the one or more images to determine that each of the additional images has additional visual data that matches the visual data in the one or more images of the object for at least one of the multiple objects, and wherein determining the current contribution of each of the multiple objects is performed at least in part based on determining that each of the additional images has additional visual data that matches the visual data in the one or more images of the object for at least one of the multiple objects.

[0180] A12. The computer-implemented method of any of clauses A01 to A11, further comprising:

[0181] comparing, by the one or more computing systems, the additional visual data of the additional images with the visual data of the one or more images to determine that one of the additional images lacks other visual data in the one or more images that matches any of the plurality of objects; and

[0182] Capturing one or more other data to provide visual data about at least one of the plurality of objects is initiated by the one or more computing systems and based on determining that the one additional image lacks other visual data in the one or more images that matches any of the plurality of objects.

[0183] A13. A computer-implemented method according to any one of clauses A01 to A12, wherein analyzing the visual data of the one or more images further comprises determining the position of each of the plurality of objects in the visual data, and wherein the method further comprises providing instructions by the one or more computing systems to capture the additional images, including providing information about the determined position of each of the plurality of objects.

[0184] A14. A computer-implemented method according to clause A13, wherein determining the position of each of the plurality of objects in the room includes: performing, by the one or more computing systems and for each of the plurality of objects, at least one of: generating a bounding box around the object in the visual data of the one or more images, or selecting pixels in the visual data of the one or more images that represent the object.

[0185] A15. A computer-implemented method according to any one of clauses A01 to A14, wherein the multiple objects installed in the room are each at least one of: a light fixture, or a sanitary ware, or a built-in furniture, or a built-in structure inside the wall of the room, or an electrical appliance, or a gas-powered appliance, or installed flooring, or installed wall covering, or installed window covering, or hardware attached to a door, or hardware attached to a window, or an installed countertop.

[0186] A16. A computer-implemented method according to any one of clauses A01 to A15, wherein analyzing the visual data of the one or more objects further identifies one or more additional objects in the room, each of which is a piece of furniture or a movable item, wherein obtaining and analyzing the visual data and determining the one or more target attributes are further performed for each of the one or more additional objects, wherein the determined assessment of the availability of the room for the purpose of indication is based on the current state of the room since the capture of the one or more images and the additional images, and wherein the method further includes determining, by the one or more computing systems, an additional assessment of the availability of the room for the purpose of indication at a later time after changing the one or more additional objects in the room and based on other images of the room captured at the later time, and providing additional information about the difference between the determined assessment since the capture of the one or more images and the additional image and the determined additional assessment at the later time.

[0187] A17. A computer-implemented method according to any one of clauses A01 to A16, wherein the assessment of the availability of the room for the indicated purpose based at least in part on combining the determined current contributions of the plurality of objects comprises performing by the one or more computing systems a weighted average of the determined current contributions of the plurality of objects, and wherein the weights used for the weighted average are based at least in part on the types of the plurality of objects.

[0188] A18. A computer-implemented method according to any of clauses A01 to A17, wherein determining the assessment of the availability of the room for the indicated purpose based at least in part on combining the determined current contributions of the plurality of objects includes providing, by the one or more computing systems, the determined current contributions of the plurality of objects to an additional trained neural network and receiving the determined assessment of the availability of the room from the additional trained neural network.

[0189] A19. A computer-implemented method according to any one of clauses A01 to A18, further comprising analyzing, by the one or more computing systems, the visual data of the one or more images to evaluate the layout of items in the room relative to the usability of the room for the indicated purpose, and wherein determining the assessment of the usability of the room for the indicated purpose is further based in part on the assessed layout of the room.

[0190] A20. A computer-implemented method according to any one of clauses A01 to A19, further comprising analyzing, by the one or more computing systems, the visual data of the one or more images to determine the type of the room and to determine the shape of the room, and determining the indicated purpose of the room based at least in part on at least one of the determined type or the determined shape, and wherein the assessment of the usability of the room for the indicated purpose is further based in part on the determined type of the room.

[0191] A21. The computer-implemented method of any of clauses A01 to A20, wherein the house comprises a plurality of rooms and has one or more associated exterior areas exterior to the house, and wherein the method further comprises:

[0192] obtaining the one or more images, analyzing the visual data, and determining the one or more target attributes, and obtaining the additional images, analyzing the additional visual data, and determining the assessment of the usability of the room, performed by the one or more computing systems and for each of the plurality of rooms;

[0193] performing, by the one or more computing systems and for each of the one or more associated external areas, obtaining one or more other images captured in the external area, and analyzing other visual data of the one or more other images to identify one or more other objects in the external area, and determining one or more other target attributes of each of the one or more other objects, and obtaining other additional images to provide other additional visual data about the one or more other target attributes of each of the one or more other objects, and analyzing the other additional visual data to determine a contribution of each of the one or more other objects to the usability of the external area for a further indication purpose, and determining an assessment of the usability of the external area for the further indication purpose based at least in part on combining the determined contributions of the one or more other objects; and

[0194] determining, by the one or more computing systems, an assessment of the overall usability of the premises based at least in part on combining information regarding the determined assessment of usability of each of the plurality of rooms with information regarding the determined assessment of usability of each of the one or more associated exterior areas, and

[0195] Wherein providing the information by the one or more computing systems further comprises displaying, by the one or more computing systems, information regarding the determined assessment of the overall usability of the premises in association with other visual information regarding the premises.

[0196] A22. A computer-implemented method according to any one of clauses A01 to A21, wherein determining the current contribution of each of the plurality of objects to the usability of the room for the indicated purpose and determining the assessment of the usability of the room for the indicated purpose are performed by evaluating the plurality of objects and characteristics of the room including at least one of condition or quality or functionality or effectiveness.

[0197] A23. A computer-implemented method according to any of clauses A01 to A22, wherein the one or more images include one or more panoramic images that, in combination, comprise 360 ​​degrees of horizontal visual coverage of the room, wherein the additional images include one or more stereo images, each stereo image having less than 180 degrees of horizontal visual coverage of the room, wherein analyzing the visual data of the one or more images and analyzing the additional visual data are performed without using any depth information from any depth sensing sensor regarding the distance from the location at which the one or more images and the additional images were captured to surrounding surfaces, wherein the additional visual data also includes at least one of a video or a three-dimensional model having visual information about at least one of the plurality of objects and / or at least one target attribute, wherein the method further includes obtaining, by the one or more computing systems, additional non-visual data having other additional details about at least one object and / or at least one target attribute, and wherein determining the current contribution of one or more of the plurality of objects is further based at least in part on analyzing the additional non-visual data and the at least one of the video or the three-dimensional model.

[0198] A24. The computer-implemented method of clause A23, wherein the plurality of images comprises one or more initial images having initial visual data and further comprises one or more additional images having additional visual data, and wherein obtaining the plurality of images comprises:

[0199] obtaining, by the one or more computing systems, the one or more initial images;

[0200] performing, by the one or more computing systems, identifying the plurality of objects based at least in part on analyzing the initial visual data of the one or more initial images;

[0201] obtaining, by the one or more computing systems, the one or more additional images to each provide additional detail regarding at least one target attribute of at least one of the plurality of objects; and

[0202] Assessing the one or more target attributes of each of the plurality of objects is performed, by the one or more computing systems, based at least in part on analyzing the additional visual data of the one or more additional images.

[0203] A25. The computer-implemented method of clause A24, further comprising:

[0204] analyzing, by the one or more computing systems, the initial visual data of the one or more initial images to determine, for each of the plurality of objects, a type of the object, and wherein determining the one or more target attributes for each of the plurality of objects is based at least in part on the determined type of the object; and

[0205] Capturing the one or more additional images is determined by the one or more computing systems based at least in part on the initial visual data of the one or more initial images lacking detail to satisfy a defined threshold for at least one target attribute for each of the plurality of objects.

[0206] A26. A computer-implemented method according to any one of clauses A01 to A25, wherein the automated operation further includes analyzing, by the one or more computing systems, the visual data of the multiple images to evaluate at least one of the layout of items in the room or the shape of the room relative to the usability of the room for the purpose of the indication, and wherein the assessment of the usability of the room for the purpose of the indication is further determined based at least in part on at least one of the assessed layout of the room or the assessed shape of the room.

[0207] A27. The computer-implemented method of any of clauses A01 to A26, wherein the building comprises a plurality of rooms and has one or more associated exterior areas exterior to the building, and wherein the automated operation further comprises:

[0208] Obtaining, analyzing, and determining the one or more target attributes and the contribution of each of the plurality of objects, and determining the assessment of the usability of the room are performed by the one or more computing systems and for each of the plurality of rooms; and

[0209] performing, by the one or more computing systems and for each of the one or more associated external areas, obtaining one or more other images captured in the external area, and analyzing other visual data of the one or more other images to identify one or more other objects in the external area, and determining one or more other target attributes of each of the one or more other objects, and determining a contribution of each of the one or more other objects to the usability of the external area for a further indication purpose, and determining an assessment of the usability of the external area for the further indication purpose based at least in part on combining the determined contributions of the one or more other objects; and

[0210] determining, by the one or more computing systems, an assessment of the overall availability of the building based at least in part on combining information regarding the determined assessment of the availability of each of the plurality of rooms with information regarding the determined assessment of the availability of each of the one or more associated exterior areas,

[0211] And wherein providing the information by the one or more computing systems further comprises displaying, by the one or more computing systems, information regarding the determined assessment of the overall usability of the building.

[0212] A28. A computer-implemented method as described in any of clauses A01 to A27, wherein the one or more images include one or more initial images having initial visual data, wherein the obtaining additional data is included in at least one of the one or more additional images or one or more videos or one or more three-dimensional models,

[0213] wherein obtaining the one or more images comprises obtaining the one or more initial images, and identifying the one or more objects is performed based at least in part on analyzing the initial visual data of the one or more initial images using at least one first trained neural network; and

[0214] Wherein evaluating the one or more target properties of each of the one or more objects is based at least in part on analyzing the additional visual data using at least one second trained neural network.

[0215] A29. A computer-implemented method according to clause A28, wherein the one or more objects include a plurality of objects, and wherein the method further comprises determining to capture the additional visual data based at least in part on a lack of detail in the initial visual data of the one or more initial images to satisfy a defined threshold for at least one target attribute with respect to each of the one or more objects.

[0216] A30. The computer-implemented method of any of clauses A01 to A29, wherein the building comprises a plurality of rooms, and wherein the automated operation further comprises:

[0217] obtaining the one or more images, and analyzing the visual data, and identifying the one or more target attributes, and obtaining the additional visual data, and determining the contribution of each of the one or more objects, and further analyzing, and determining the assessment of the usability of the room, performed by the one or more computing systems and for each of the plurality of rooms; and

[0218] determining, by the one or more computing systems, an assessment of overall availability of the building based at least in part on combining information of the determined assessments of availability of each of the plurality of rooms, and

[0219] Wherein providing the information further comprises displaying, by the one or more computing systems, information regarding the determined assessment of the overall usability of the building.

[0220] A31. A computer-implemented method comprising a plurality of steps to perform automated operations implementing the described techniques substantially as disclosed herein.

[0221] B01. A non-transitory computer-readable medium storing executable software instructions and / or other stored content, wherein the executable software instructions and / or other stored content cause one or more computing systems to perform automated operations, wherein the automated operations implement the method described in any one of clauses A01 to A31.

[0222] B02. A non-transitory computer-readable medium storing executable software instructions and / or other stored content that causes one or more computing systems to perform automated operations that implement the described techniques substantially as disclosed herein.

[0223] C01. One or more computing systems comprising one or more hardware processors and one or more memories storing instructions, wherein the instructions, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations, wherein the automated operations implement the method described in any one of clauses A01 to A31.

[0224] C02. One or more computing systems comprising one or more hardware processors and one or more memories storing instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations that implement the described techniques substantially as disclosed herein.

[0225] D01. A computer program adapted to perform the method of any one of clauses A01 to A31 when the computer program is run on a computer.

[0226] This article describes aspects of the present disclosure with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart illustration and / or block diagram and the combination of blocks in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions. It will be further understood that in some embodiments, the functions provided by the routines discussed above can be provided in an alternative manner, such as being divided into more routines or being merged into fewer routines. Similarly, in some embodiments, the routines shown can provide more or less functions than the functions described, such as when other routines shown are correspondingly changed to lack or include such functions, or when the amount of the functions provided changes. In addition, although various operations can be described as being performed in a specific manner (e.g., serial or parallel or synchronous or asynchronous) and / or in a specific order, in other embodiments, operations can be performed in other orders and in other ways. Any data structure discussed above can also be structured in different ways, such as by dividing a single data structure into multiple data structures and / or by merging multiple data structures into a single data structure. Similarly, in some embodiments, illustrated data structures may store more or less information than described, such as when other illustrated data structures are correspondingly altered to lack or include such information, or when the amount or type of stored information is varied.

[0227] In light of the foregoing, it will be understood that although specific embodiments have been described herein for illustrative purposes, various modifications may be made without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited except by the corresponding claims and the elements recited in those claims. Furthermore, while certain aspects of the present invention may be presented in certain claim forms at certain times, the inventors contemplate various aspects of the present invention in any available claim form. For example, while only some aspects of the present invention may be described as embodied in a computer-readable storage medium at a particular time, other aspects may also be embodied in this manner.

Claims

1. A computer-implemented method comprising: Obtaining, by one or more computing systems, one or more images captured at one or more first acquisition locations in a room of a house, wherein the one or more images are panoramic images having wide-angle visual coverage; analyzing, by the one or more computing systems, visual data of the one or more images to identify a plurality of objects installed in the room and, for each of the plurality of objects, determine at least a type of the object, the analyzing comprising: determining, by the one or more computing systems and for each of the plurality of objects, one or more target properties of the object based at least in part on the type of the object, and determining an amount of information in the visual data about the one or more target properties of each of the plurality of objects; and determining, by the one or more computing systems, to capture additional images to each provide additional visual data having additional detail regarding at least one target attribute of at least one of the plurality of objects, wherein the determination to capture the additional images is based at least in part on a determination that an amount of information in the visual data is insufficient for each of the plurality of objects to allow evaluation of the one or more target attributes of each of the plurality of objects such that a defined detail threshold for the one or more target attributes of the object is not satisfied; obtaining, by the one or more computing systems and based at least in part on determining to capture the additional images, the additional images captured in the room to each provide the additional visual data having additional detail regarding at least one target attribute of at least one of the plurality of objects, wherein the additional images are captured at one or more second capture locations that are closer to at least one object than the one or more first capture locations, and wherein the additional images are more focused than the one or more images and are one or more stereo images in a rectilinear format; analyzing, by the one or more computing systems and via use of at least one trained neural network, the additional visual data of the additional images to evaluate the plurality of objects, including determining, for each of the plurality of objects, a current contribution of the object to the usability of the room for the indicated purpose based at least in part on assessing the one or more target attributes of the object using information determined from the additional visual data; determining, by the one or more computing systems and based at least in part on combining the determined current contributions of the plurality of objects, an assessment of the availability of the room for the indicated purpose; and Information regarding the determined assessment of the availability of the room is provided, by the one or more computing systems.

2. The computer-implemented method of claim 1 , wherein: Analyzing the visual data of the one or more images to identify the plurality of objects further comprises identifying one or more additional objects in the room separate from the plurality of objects, and wherein the method further comprises: analyzing, by the one or more computing systems, the visual data of the one or more images to determine a further amount of information in the visual data regarding one or more additional target attributes of each of the one or more additional objects; and Determining, by the one or more computing systems, not to capture further additional images based, at least in part, on determining that a further amount of information in the visual data for each of the one or more additional objects is sufficient to satisfy a defined detail threshold for the one or more additional target attributes of the additional object.

3. The computer-implemented method of claim 1 , further comprising at least one of the following steps: analyzing, by the one or more computing systems, the additional visual data of the additional image to determine a further amount of information in the additional visual data regarding the one or more target attributes of each of the plurality of objects; and determining, by the one or more computing systems, that the further amount of information in the additional visual data is sufficient to satisfy a defined detail threshold for the one or more target attributes for each of the plurality of objects, and wherein determining the current contribution of each of the plurality of objects is performed based at least in part on determining that the further amount of information in the additional visual data is sufficient to satisfy the defined detail threshold for the one or more target attributes for each of the plurality of objects; or analyzing, by the one or more computing systems, the additional visual data of the additional image to determine a further amount of information in the additional visual data regarding the one or more target attributes of each of the plurality of objects; determining, by the one or more computing systems, and for one of the plurality of objects, that the further amount of information in the additional visual data is insufficient to satisfy a defined detail threshold for at least one target attribute of the one object; and initiating, by the one or more computing systems, and based on determining that the further amount of information in the additional visual data is insufficient to satisfy the defined detail threshold for at least one target attribute of the one object, and before determining the current contribution of the one object, capturing one or more additional images of the one object to provide additional visual data sufficient to satisfy the defined detail threshold for the one or more target attributes of the one object.

4. The computer-implemented method of claim 1 , further comprising: The additional visual data of the additional images is compared with the visual data of the one or more images by the one or more computing systems to determine that each of the additional images has additional visual data of at least one of the multiple objects that matches the visual data in the one or more images of the object, and wherein, if it has been determined that each of the additional images has additional visual data of at least one of the multiple objects that matches the visual data in the one or more images of the object, determining the current contribution of each of the multiple objects is performed.

5. The computer-implemented method of claim 1 , further comprising: comparing, by the one or more computing systems, the additional visual data of the additional images with the visual data of the one or more images to determine that one of the additional images lacks visual data that matches other visual data in the one or more images of any of the plurality of objects; as well as Capturing one or more other data is initiated, by the one or more computing systems, and based on a determination that the one additional image lacks visual data that matches other visual data in the one or more images of any of the multiple objects, to provide additional visual data regarding at least one of the multiple objects.

6. The computer-implemented method of claim 1 , wherein: Analyzing the visual data of the one or more images also includes determining the location of each of the multiple objects in the visual data, and wherein the method further includes: providing instructions, via the one or more computing systems, to capture the additional images, including providing information about the determined location of each of the multiple objects.

7. The computer-implemented method of claim 1 , wherein: Each of the multiple objects installed in the room is at least one of the following: a light fixture, or a plumbing fixture, or a built-in furniture, or a built-in structure inside the wall of the room, or an electrical appliance, or a gas-powered appliance, or installed flooring, or installed wall covering, or installed window covering, or hardware attached to a door, or hardware attached to a window, or an installed countertop.

8. The computer-implemented method of claim 1 , wherein: Analyzing the visual data of the one or more images further identifies one or more additional objects in the room, each of which is a piece of furniture or a movable item, wherein for each of the one or more additional objects, further performing: obtaining and analyzing the visual data, wherein the determined assessment of the availability of the room for the purpose of indication is based on the current state of the room since the one or more images and the additional images were captured, and wherein the method further includes: determining, by the one or more computing systems, an additional assessment of the availability of the room for the purpose of indication at a later time after changing the one or more additional objects in the room and based on other images of the room captured at the later time, and providing additional information about the difference between the determined assessment since the capture of the one or more images and the additional image and the determined additional assessment at the later time.

9. The computer-implemented method of claim 1 , wherein: Determining the assessment of the availability of the room for the indicated purpose based at least in part on combining the determined current contributions of the plurality of objects includes at least one of: performing, by the one or more computing systems, a weighted average of the determined current contributions of the plurality of objects, and wherein weights for the weighted average are based at least in part on the types of the plurality of objects; or The determined current contributions of the plurality of objects are provided to an additional trained neural network by the one or more computing systems, and the determined assessment of the availability of the room is received from the additional trained neural network.

10. The computer-implemented method of claim 1 , further comprising: The visual data of the one or more images is analyzed, by the one or more computing systems and by using at least one trained neural network, to determine a type of the room and to determine a shape of the room, and to determine an indicated purpose of the room based at least in part on at least one of the determined type or the determined shape, and wherein determining the assessment of the usability of the room for the indicated purpose is further based in part on the determined type of the room.

11. The computer-implemented method of claim 1 , further comprising: performing, by the one or more computing systems, and for each of a plurality of rooms of the house: obtaining the one or more images, analyzing the visual data, obtaining the additional images, analyzing the additional visual data, and determining the assessment of the usability of the room; performing, by the one or more computing systems, and for each of one or more associated exterior areas of the exterior of the premises, obtaining one or more other images captured in the exterior area, and analyzing other visual data of the one or more other images to identify one or more other objects in the exterior area, and determining one or more other target attributes of each of the one or more other objects, and obtaining other additional images to provide other additional visual data about the one or more other target attributes of each of the one or more other objects, and analyzing the other additional visual data to determine a contribution of each of the one or more other objects to the usability of the exterior area for a further indication purpose, and determining, based at least in part on combining the determined contributions of the one or more other objects, an assessment of the usability of the exterior area for the further indication purpose; as well as determining, by the one or more computing systems, an assessment of the overall usability of the premises based at least in part on combining information regarding the determined assessment of usability of each of the plurality of rooms with information regarding the determined assessment of usability of each of the one or more associated exterior areas, and wherein providing, via the one or more computing systems, the information further comprises displaying, via the one or more computing systems, information regarding the determined assessment of the overall usability of the premises in association with other visual information regarding the premises.

12. The computer-implemented method of claim 1 , wherein: Determining the current contribution of each of the plurality of objects to the usability of the room for the indicated purpose, and determining the assessment of the usability of the room for the indicated purpose are performed by evaluating the plurality of objects and characteristics of the room including at least one of condition, or quality, or functionality, or effectiveness.

13. The computer-implemented method of claim 1 , wherein: The one or more images taken together comprise 360 ​​degrees of horizontal visual coverage of the room, wherein the additional images each have less than 180 degrees of horizontal visual coverage of the room, wherein analyzing the visual data of the one or more images and analyzing the additional visual data are performed without using any depth information from any depth sensing sensor regarding the distance to surrounding surfaces from the location at which the one or more images and the additional images were captured, wherein the additional visual data also includes at least one of a video or a three-dimensional model having visual information about at least one of the plurality of objects and / or at least one target attribute, wherein the method further comprises: obtaining, by the one or more computing systems, additional non-visual data having other additional details about at least one object and / or at least one target attribute, and wherein determining the current contribution of one or more of the plurality of objects is further based in part on analyzing at least one of the video or the three-dimensional model and the additional non-visual data.

14. A non-transitory computer-readable medium storing content that causes one or more computing devices to perform automated operations, the automated operations comprising at least: obtaining, by the one or more computing devices, a plurality of images captured in a room of a building, wherein the plurality of images includes one or more initial panoramic images captured at one or more first acquisition positions, and wherein the initial panoramic images have a wide-angle visual coverage showing a plurality of objects in the room; analyzing, by the one or more computing devices and using at least one trained neural network, visual data from the plurality of images to identify a plurality of objects in the room, including determining an amount of information in the visual data about one or more target attributes of each of the plurality of objects; determining, by the one or more computing systems, to capture one or more additional images to each provide additional visual data having additional detail regarding at least one target attribute of at least one of the plurality of objects, wherein the determination to capture the additional images is based at least in part on a determination that an amount of information in the visual data for each of the plurality of objects is insufficient to allow evaluation of the one or more target attributes of each of the plurality of objects such that a defined detail threshold for the one or more target attributes of the object is not satisfied; capturing, by the one or more computing devices and based at least in part on determining, the one or more additional images, obtaining the one or more additional images, the one or more additional images captured at one or more second capture locations in the room that are closer to at least one object of the plurality of objects than the one or more first capture locations, and providing the additional visual data having the additional details regarding the at least one object; determining, by the one or more computing devices, and for each of the plurality of objects, one or more target attributes of the object, and determining a contribution of the object to the usability of the room for purposes of indication based at least in part on assessing the one or more target attributes of the object using at least one of the visual data of the plurality of images and additional visual data of the one or more additional images; determining, by the one or more computing devices and based at least in part on combining the determined contributions of the plurality of objects, an assessment of the availability of the room for the indicated purpose; and Information regarding the determined assessment of the availability of the room is provided, by the one or more computing devices.

15. A system comprising: one or more hardware processors of one or more computing systems; as well as One or more memories storing instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing systems to perform automated operations, the automated operations comprising at least: obtaining one or more images captured at one or more first acquisition locations in a room of a building, wherein the one or more images are panoramic images having wide-angle visual coverage; analyzing visual data of the one or more images to identify one or more objects in the room and determine a type of each of the one or more objects, including: identifying one or more target attributes of each of the one or more objects based at least in part on the determined type of the objects, and determining an amount of information in the visual data about the one or more target attributes of each of the one or more objects; determining to capture additional images in the room to each provide additional visual data having additional detail regarding at least one target attribute of at least one of the one or more objects, wherein the determination to capture the additional images is based at least in part on a determination that an insufficient amount of information in the visual data for each of the one or more objects is sufficient to allow evaluation of the one or more target attributes of each of the one or more objects such that a defined detail threshold for the one or more target attributes of the objects is not satisfied; obtaining the additional visual data having the additional details about at least one target attribute of at least one of the one or more objects for the additional images in the room, wherein the additional images are captured at one or more second acquisition positions closer to the at least one object than the one or more first acquisition positions, and wherein the additional images are more focused than the one or more images and are one or more stereo images in a rectilinear format; determining, for each of the one or more objects, a contribution of the object to the usability of the room based at least in part on an assessment of the one or more target properties of the object, wherein the assessment is based at least in part on the additional visual data; further analyzing at least one of the visual data or the additional visual data using at least one trained neural network to assess at least one of a shape of the room or a layout of items in the room; determining the assessment of the usability of the room based at least in part on combining information regarding the determined contributions of the one or more objects with information regarding at least one of an assessed shape or layout of the room; and Information regarding the determined assessment of the availability of the room is provided.

Images