Automatic room shape determination using visual data of multiple captured indoor images

Abstract

A technique is described for automated operations to analyze a combination of multiple images captured from a room to determine a room shape, such as by iteratively refining alignment of visual data of the multiple images into a common coordinate system until alignment differences meet one or more defined criteria positions, and then using the determined room shape information in a further automated manner. The images can be equirectangular or other spherical format panorama images, and the room shape determined for one or more rooms of a building can be a fully enclosed three-dimensional shape and used for improved navigation of the building (e.g., as part of a generated building floor plan), the automated room shape determination can be further performed without having or using information from any ranging device about distances from a capture position of an image to walls or other objects in a surrounding room.

Description

Technical Field

[0001] The following disclosure generally relates to the following techniques: for automatically analyzing visual data from a combination of multiple images captured in a room of a building to determine the shape of the room, and for subsequently using the determined room shape information in one or more ways, such as for iteratively refining the alignment of the visual data of the multiple images until a three-dimensional room shape of a fully closed flat surface is generated from the aligned combination of visual data of the multiple images, and for using the determined room shape to improve navigation of the building. Background Technology

[0002] In various fields and situations (such as building analysis, property inventory, real estate acquisition and development, renovation and alteration services, general contracting, and others), it may be desirable to view information about the interior of a house, office, or other building without having to physically visit or enter the building. This includes determining actual as-built information about a building rather than design information obtained before construction. However, it can be difficult to effectively capture, represent, and use such building interior information, including displaying visual information captured inside the building to users at remote locations (e.g., enabling users to fully understand the interior layout and other details, including user-selectable control of the display). Furthermore, while floor plans can provide information about the layout and some aspects of the building's interior, using floor plans in this way has several drawbacks in certain situations, including the difficulty in constructing and maintaining floor plans, the difficulty in accurately scaling and filling in information about room interiors, and the difficulty in visualizing and otherwise using them. Attached Figure Description

[0003] Figures 1A to 1B The diagram depicts an exemplary building environment and computing system used in embodiments of this disclosure, such as performing automated operations to iteratively combine visual data from multiple images captured in a room to determine the room shape and then using the determined room shape information in one or more ways.

[0004] Figures 2A to 2V An example of an automated operation is shown that determines the shape of a building’s rooms by iteratively combining visual data from multiple images captured in the room and then uses the determined room shape information in one or more ways, including information for generating and presenting floor plans of the building.

[0005] Figure 3 This is a block diagram illustrating a computing system suitable for implementing one or more systems that perform at least some of the techniques described in this disclosure.

[0006] Figure 4An exemplary flowchart of an image capture and analysis (ICA) system routine according to an embodiment of the present disclosure is shown.

[0007] Figures 5A to 5C An exemplary flowchart of a Mapping Information Generation Manager (MIGM) system routine according to an embodiment of the present disclosure is shown.

[0008] Figure 6 An exemplary flowchart of a building map viewer system routine according to an embodiment of the present disclosure is shown. Detailed Implementation

[0009] This disclosure describes techniques for using computing devices to perform and analyze visual data from combinations of multiple images captured in rooms of a building to determine the shape of the rooms and for subsequently using the determined room shape information in one or more further automated ways. For example, the images may include panoramic images (e.g., in isometric or other spherical formats) and / or other types of images (e.g., in linear perspective formats) captured at or around a multi-room building (e.g., a house, office, etc.), generally referred to herein as “target images.” Furthermore, in at least some of these embodiments, automated room shape determination is further performed without requiring or using information from any depth sensor or other ranging device regarding distances from the acquisition location of the target image to walls or other objects in the surrounding building. In various embodiments, the determined room shape information for one or more rooms of a building may be further used in various ways, such as in conjunction with generating corresponding building floor plans and / or other generated mapping-related information about the building, including for controlling navigation of mobile devices (e.g., autonomous vehicles), for displaying or otherwise presenting in a corresponding GUI (graphical user interface) on one or more client devices via one or more computer networks, etc. Additional details regarding the automated determination and use of room shape information are included below, and in at least some embodiments, some or all of the techniques described herein can be performed via automated operation of a Mapping Information Generation Manager (“MIGM”) system, as further discussed below.

[0010] As described above, the automated operation of the MIGM system may include determining the shape of the room based on the analysis of visual data from a combination of multiple target images captured in the room (such as multiple panoramic images captured at multiple acquisition locations in the room). In at least some embodiments, each of the multiple panoramic images includes a 360° horizontal visual coverage around a vertical axis and a visual coverage of some or all of the floor and / or ceiling in the room (e.g., 180° or more vertical visual coverage), and each has an isorectangular format or other spherical format (e.g., represented in a spherical coordinate system, where vertical lines in the surrounding environment are straight in the image, and where horizontal lines in the surrounding environment (which curve further away from the vertical midpoint of the image) gradually curve away from the horizontal in a convex manner in the image). In at least some embodiments, the automated operation of the MIGM system includes performing iterative operations to finely align visual data of multiple target images in a room to a common coordinate system, such as until the difference in visual data alignment is below a defined threshold or otherwise meets one or more defined criteria, and then using the aligned combined visual data of the multiple target images to generate a three-dimensional (“3D”) room shape, which may use flat surfaces representing the walls, floors and ceiling of the room, which are fully connected to form a closed 3D geometry, and in at least some embodiments also includes indications of the location and shape of windows and / or doorways and / or other wall openings.

[0011] As part of the automated iterative operation of the MIGM system, to finely align the visual data of multiple target panoramic images captured in a room to a common coordinate system, the MIGM system can use initial pose information to generate one or more linearly formatted projected perspective images via a camera device or other image acquisition device to acquire target panoramic images. Each projected perspective image includes a subset of the visual data of the target panoramic image, such as a linearly formatted perspective image including visual data of some or all of the floor and portions of the walls connected to the floor (sometimes referred to herein as a “floor view image” or “floor image”), and / or a linearly formatted perspective image including visual data of some or all of the ceiling and portions of the walls connected to the ceiling (sometimes referred to herein as a “ceiling view image” or “ceiling image”). Such pose information of the target panoramic image may include the acquisition position of the target panoramic image within the room (e.g., in three dimensions or degrees of freedom, and sometimes represented in a 3D mesh as an X, Y, Z tuple) and the orientation of the target panoramic image (e.g., in three additional dimensions or degrees of freedom, and sometimes represented as a 3D rotation tuple or other direction vector), and is sometimes also referred to herein as the “acquisition pose” or “acquisition position” or simply “position” of the target panoramic image. In some embodiments, the image acquisition device for capturing panoramic images of a target may be a mobile computing device that includes one or more cameras or other imaging systems (optionally including one or more fisheye lenses) and optionally includes additional hardware sensors for capturing non-visual data, such as one or more inertial measurement units (or “IMU”) sensors that capture data reflecting the motion of the device. In other embodiments, it may be a camera device that lacks computing power and is optionally associated with a nearby mobile computing device.

[0012] After generating one or more projected perspective images from the target panoramic image, the automated operation of the MIGM system also includes analyzing the visual content of each target panoramic image and its projected perspective image to identify elements visible in the visual content (e.g., two-dimensional or 2D elements), such as identifying structural elements of the walls and floors and ceilings of the surrounding room (e.g., windows and / or skylights; passageways into and / or out of the room, such as doorways and other openings in the walls, stairs, corridors, etc.; boundaries between adjacent walls; boundaries between walls and floors; boundaries between walls and ceilings; boundaries between floors and ceilings; corners (or solid geometric vertices) where at least three surfaces or planes intersect, etc.), and optionally identifying other fixed structural elements (e.g., countertops, bathtubs, sinks, islands, fireplaces, etc.), and optionally generating 3D bounding boxes of the identified elements or otherwise tracking the position of the identified elements. Automated analysis of visual data for each target panoramic image and its projected perspective image may also include generating an estimated partial room shape from the visual data and optionally from additional data captured during or near the acquisition of the panoramic image (e.g., IMU motion data from the image acquisition device and / or the accompanying mobile computing device), and determining the location of the identified elements within the estimated partial room shape. In at least some of these implementations, the partial room shape estimated from the target panoramic image and its projected perspective image may be a 3D point cloud (having multiple 3D data points corresponding to locations on the walls and optionally on the floor and / or ceiling) and / or a broken partial planar surface (corresponding to portions of the walls and optionally the floor and / or ceiling) and / or wireframe structural lines (e.g., showing one or more boundaries between walls, boundaries between walls and ceilings, boundaries between walls and floors, outlines of doorways and / or other wall openings between rooms, outlines of windows, etc.), which is at least in part based on performing SLAM (Simultaneous Localization and Mapping) and / or SfM (Structure Retrieval in Motion) and / or MVS (Multi-View Stereo Vision) analysis (e.g., using motion data from IMU sensors of an image acquisition device and / or an associated nearby mobile computing device in the same room, combined with visual data from one or more image sensors of the image acquisition device) and / or ICP (Iterative Nearest Point) analysis. In at least some implementations, the generated projected perspective image may be further manipulated, such as by scaling and rotating corresponding areas in the corresponding image (e.g., the layout of the floor and ceiling), which have additional details about the generation of the projected perspective image, as described below.

[0013] After generating each target panoramic image by analyzing visual data from each target panoramic image (including visual data from target panoramic images contained in one or more projected perspective images), the automated operation of the MIGM system further includes analyzing the differences between the generated information of each target panoramic image in the room and determining whether the differences are sufficiently small (e.g., below a defined threshold, or otherwise meet one or more defined criteria). If the differences are not small enough, the automated operation further includes adjusting the pose information of at least some target panoramic images based on the difference information (in order to reduce or eliminate the differences), and performing the next iterative analysis by projecting one or more new perspective images of each of these target panoramic images and by identifying elements and their positions in the new perspective images of these target panoramic images, and optionally determining new estimated partial room shapes based partly on the visual data in those new perspective images and by determining the difference information based partly on the identification information in those new perspective images, wherein the iterative process continues until the determined differences are sufficiently small.

[0014] In at least some embodiments, pose information adjustment is performed on all target panoramic images acquired in the room, except for one target panoramic image selected as the first target image or reference target image, and the adjusted pose information for those other second target panoramic images is determined relative to the initial pose information of the first target image / reference target image. Difference analysis can be performed, for example, using one or more trained convolutional neural networks, such as neural networks trained to identify differences in the determined positions of identified elements in different target panoramic images and / or differences in the layout of the floor and / or ceiling in projected perspective images from different target panoramic images. If differences are identified in the determined positions of identified elements between the first target image / reference panoramic image and another second target panoramic image, such as positional offsets at least in part based on the identified features themselves and / or the determined bounding boxes of the identified features, the output of one or more trained convolutional neural networks can include changes in the position and orientation (e.g., rotation) of some or all of the identified elements from the visual data of the two target panoramic images, these changes can be aggregated and then used to adjust the current pose information (in the first iteration, the initial pose information) for the other second target panoramic images. If differences are identified in the layout of the floor and / or ceiling based on visual data of a first panoramic image / reference panoramic image (e.g., at least in part based on one or more projected perspective images of the first panoramic image) and visual data of another second target panoramic image (e.g., at least in part based on one or more current projected perspective images of the second panoramic image from the most recent iteration analysis), such as offsets in the positions of corresponding portions of the floor and / or ceiling layout at least in part, the output of one or more trained convolutional neural networks may include variations in the floor and / or ceiling layout in the visual data of the second target panoramic image to match the floor and / or ceiling layout in the visual data of the first target panoramic image / reference target panoramic image, wherein new pose information determined for the second target panoramic image will result in a changed layout (e.g., by performing differentiable rendering analysis and using backpropagation to directly adjust the current pose information of the other second target panoramic image, which was the initial pose information of the second target panoramic image in the first iteration).

[0015] If the difference analysis indicates that the current pose information after one or more iterations of one or more second target panoramic images produces a sufficiently small difference, the automated operation of the MIGM system also includes using information from the analysis of visual data from the target panoramic images in the room to determine the final room shape. Specifically, these further automated operations of the MIGM system may include using information from the analysis of visual data from a first target panoramic image / reference target panoramic image (including information from the visual data of the first target panoramic image / reference target panoramic image, which includes, in one or more projected perspective images, such as identified elements and their determined locations, and optionally a determined estimated partial room shape) and from the analysis of the most recent iteration of visual data from each second target panoramic image (including visual data from the second target panoramic image included in one or more recently projected perspective images for the most recent iteration, such as identified elements and their determined locations, and optionally a determined estimated partial room shape), such as feeding information from the analysis of visual data from multiple target panoramic images to one or more trained convolutional neural networks for further analysis. In at least some embodiments, the output of one or more trained convolutional neural networks includes a fully enclosed 3D room shape comprising fully connected planar surfaces for each wall and at least one of the floor and ceiling (e.g., both the floor and ceiling perspective images are projected for at least some target panoramic images).

[0016] The following includes additional details about the automated operation of the MIGM system for determining the shape of a room based on the analysis of combined visual data from multiple target panoramic images captured in the room.

[0017] Furthermore, in some implementations, the automated operation of the MIGM system includes obtaining one or more types of input information from one or more users (e.g., system operator users of the MIGM system assisting in its operation, end users obtaining information results from the MIGM system, etc.) and using this input information to facilitate certain operations of the MIGM system. As a non-exclusive example, user-provided input may include one or more of the following: input specifying initial pose information for a target panoramic image used in a first iteration or initial automated determination of pose information for a refined target panoramic image used in a first iteration; input specifying adjusted pose information for a target panoramic image used in a later iteration or initial automated determination of pose information for a refined target panoramic image used in a later iteration; input specifying initial automated identification of elements in the visual data of a target panoramic image (including in one or more projected perspective images from the target panoramic image) or in the visual data of a refined target panoramic image; input specifying initial automated determination of the position of elements in the visual data of a target panoramic image (including in one or more projected perspective images from the target panoramic image) or in the visual data of a refined target panoramic image. The input can be used to specify an estimated partial room shape from visual data of a target panoramic image (including from one or more projected perspective images of the target panoramic image), or to refine the initial automated generation of such an estimated partial room shape from visual data of a target panoramic image; the input can be used to specify a final room shape by merging or otherwise combining estimated partial room shapes based on visual data from multiple target panoramic images (including projected perspective images from those target panoramic images), or to refine the initial automated generation of such a final room shape by such merging or other combination; the input can be used to specify a final room shape based on a combination of visual data from multiple target panoramic images (including projected perspective images from those target panoramic images), or to refine the initial automated generation of such a final room shape based on a combination of visual data from multiple target panoramic images; and so on. Furthermore, in some implementations and cases, user-provided input can be incorporated into subsequent automated analysis in various ways, including replacing or supplementing automatically generated information of the same type to be used as constraints and / or prior probabilities in later automated analysis processes (e.g., via trained neural networks).

[0018] Furthermore, in some implementations, the automated operation of the MIGM system also includes acquiring and using additional types of information during its analysis activities. Non-exclusive examples of the use of such additional types of information include: acquiring and using the name or other labels of a specific room, such as grouping images acquired in the same room; acquiring initial pose information, such as from automated analysis that may include SLAM and / or SfM and / or MVS, to serve as the target panoramic image during the first iteration; acquiring and using other image acquisition metadata to group images or otherwise assist image analysis, such as using image acquisition time information and / or sequence information to identify consecutive images that may have been captured in the same room; and so on. Additional details regarding other automated operations of the MIGM system in some implementations and situations are included below.

[0019] The described technology offers various benefits in various implementations, including allowing the automatic generation of partial or complete floor plans of multi-room buildings and other structures from target images acquired for buildings or other structures, including providing more complete and accurate room shape information and a wider variety of environmental conditions (e.g., in cases where objects in the room obscure the view of a single image of at least some walls and / or floors and / or ceilings, but where the combination of views from multiple images eliminates or reduces the problem, in cases where the color and texture of at least some wall and / or ceiling and / or floor surfaces are almost unchanged and lack extensive visible 3D features, etc.), and in some implementations including the absence or non-use of information from depth sensors or other ranging devices regarding distances from the image acquisition location to other objects in the walls or surrounding buildings or other structures. Non-exclusive examples of the additional benefits of the technology include: analyzing visual data of target images to detect objects of interest (e.g., structural wall elements, such as windows, doorways, and other wall openings) in an enclosed room and determining the positions of those detected objects within a defined room shape; analyzing additional captured data (e.g., motion data from one or more IMU sensors, visual data from one or more image sensors, etc.) to determine the path of the image acquisition device in multiple rooms; identifying wall openings (e.g., doorways, staircases, etc.) in multiple rooms, at least in part based on the additional data (and optionally based on visual data from one or more target images acquired in one or more rooms); and optionally further using information about the identified walls. Such information about wall openings allows for the positioning of a defined 3D room shape for multiple rooms; the ability to interconnect multiple target images and display at least one of the target images with user-selectable visual indicators in the direction of other linked target images, such that, when selected, a corresponding other linked target image is displayed, such as by placing the various target images in a common coordinate system displaying their relative positions, or otherwise determining at least the orientation between pairs of target images (e.g., at least in part based on automated analysis of the visual content of the target images in the pair, and optionally based on additional movement data from a mobile computing device along the path of travel between the target images), and linking the individual target images using the determined inter-image orientation; and so on. Furthermore, the described automation techniques allow for faster determination of such room shape information than previously existing techniques and have higher accuracy in at least some embodiments, including by using information acquired from the actual building environment (rather than from plans about how the building should theoretically be constructed), and enabling the capture of changes in structural elements that occur after the initial construction of the building.The described technique further offers benefits, at least in part, based on the determined image acquisition location, allowing for improved automated navigation of buildings by mobile devices (e.g., semi-autonomous or fully autonomous vehicles), including a significant reduction in computational power and time spent attempting to otherwise learn the building's layout. Additionally, in some embodiments, the described technique can be used to provide an improved GUI where users can obtain information more accurately and quickly about the building's interior and / or other associated areas (e.g., for use when navigating within that interior), including in response to search requests, as part of providing personalized information to users, as part of providing value estimates and / or other information about the building, etc. The described technique also provides a variety of other benefits, some of which are further described elsewhere herein.

[0020] As described above, in at least some implementations and situations, some or all of the images captured for a building can be panoramic images, each captured at one of a plurality of capture locations inside or around the building, such as panoramic images of each such capture location to generate one or more videos captured at that capture location (e.g., 360° video taken from a user-held smartphone or other mobile device rotating at that capture location), or multiple images captured from the capture location in multiple directions (e.g., from a user-held smartphone or other mobile device rotating at that capture location; from automatic rotation of a device at that capture location, such as on a tripod at that capture location; etc.), or all image information of a particular capture location can be captured simultaneously (e.g., using one or more fisheye lenses), etc. It should be understood that such panoramic images may, in some cases, be represented in a spherical coordinate system and provide coverage of up to 360° around a horizontal and / or vertical axis (e.g., 360° coverage along a horizontal plane and around a vertical axis), while in other embodiments, the acquired panoramic or other images may include a vertical coverage of less than 360° (e.g., for images whose width exceeds their height and whose aspect ratio exceeds a typical aspect ratio such as 21:9 or 16:9 or 3:2 or 7:5 or 4:3 or 5:4 or 1:1, including so-called “ultra-wide” lenses and the resulting ultra-wide images). Additionally, it will be understood that users who are permitted to view such a panoramic image (or other image with sufficient horizontal and / or vertical coverage to display only a portion of the image at any given time) may move their viewing orientation within the panoramic image to different orientations to indicate that different subsets of the image (or "views") are rendered within the panoramic image, and in some cases, such a panoramic image may be represented in a spherical coordinate system (including, if the panoramic image is represented in a spherical coordinate system and a particular view is rendered, converting the image to a planar coordinate system, such as for a perspective image view prior to display). Furthermore, acquisition metadata regarding the capture of such panoramic images can be obtained and used in various ways, such as data acquired from IMU sensors or other sensors of the mobile device while it is carried by a user or otherwise moved between acquisition locations. Non-exclusive examples of such acquisition metadata may include one or more of the following: acquisition time; acquisition location, such as GPS coordinates or other location indications; acquisition direction and / or orientation; relative or absolute acquisition order of multiple images acquired for or otherwise associated with buildings; etc. In at least some implementations and situations, such acquisition metadata may also optionally be used as part of determining the acquisition location of the image, as discussed further below. Additional details are included below regarding the automated operations involved in acquiring images and optionally acquiring metadata in the implementation of an Image Capture and Analysis (ICA) system, including information on… Figures 1A to 1B and Figures 2A to 2DAnd additional details elsewhere in this article.

[0021] As described above, in at least some embodiments, a building floor plan with associated room shape information for some or all of the building's rooms can be generated and further used in one or more ways, such as in the subsequent automated determination of image acquisition locations within the building. In various embodiments, the building floor plan with associated room shape information can take various forms, such as a 2D (two-dimensional) floor map of the building (e.g., an orthogonal top view or other plan view of a schematic floor map excluding or displaying height information) and / or a 3D (three-dimensional) or 2.5D (two-and-a-half-dimensional) floor map model of the building displaying height information. Furthermore, in various embodiments, the shapes of the building's rooms can be automatically determined in various ways, including in some embodiments, time-based determination prior to the automated determination of acquisition locations for specific images within the building. For example, in at least some implementations, a Mapping Information Generation Manager (MIGM) system can analyze various images acquired inside and around a building to automatically determine the room shapes of the building's rooms (e.g., 3D room shapes, 2D room shapes, etc., such as reflecting the geometry of the building's surrounding structural elements). This analysis may include, for example, automated operations to "register" the camera positions of the images in a common reference frame to "align" the images, and to estimate the 3D positions and shapes of objects in the rooms, such as by determining features visible in the content of such images (e.g., determining the orientation and / or orientation of the acquisition device when capturing a particular image, the path the acquisition device travels through the room, etc., such as by using SLAM technology for multiple views). The process involves using frame images and / or other SfM techniques to create a set of “dense” images spaced at a maximum defined distance (such as 6 feet) to generate a 3D point cloud, which includes 3D points along at least some of the walls, ceiling, and floor of the room and optionally 3D points corresponding to other objects in the room, etc.) and / or identifying possible locations of the walls and other surfaces of the room by determining and aggregating information about the planes of the detected features and the normal (or orthogonal) directions of which planes, and connecting various possible wall locations (e.g., using one or more constraints, such as having a 90° angle between walls and / or between walls and the floor, as part of the so-called “Manhattan world hypothesis”) and forming an estimated partial room shape of the room. After determining the estimated partial room shapes of the rooms in the building, in at least some embodiments, the automated operation may further include positioning multiple room shapes together to form a floor plan of the building and / or other relevant mapping information, such as connecting the various room shapes by optionally at least in part based on information about doorways and stairwells and wall openings between other rooms identified in a particular room and optionally at least in part based on determined travel path information of a mobile computing device between rooms.The following includes additional details regarding the automated operations involved in implementing the MIGM system in determining and combining room shapes to generate floor plans, including details about... Figures 1A to 1B and Figures 2E to 2V And additional details elsewhere in this article.

[0022] For illustrative purposes, some embodiments are described below in which specific types of information are acquired, used, and / or presented in a specific manner and by using specific types of devices for a particular type of structure. However, it will be understood that the described techniques can be used in other ways in other embodiments, and therefore the invention is not limited to the exemplary details provided. As a non-exclusive example, although a floor plan that does not include detailed measurements of specific rooms or the entire house can be generated for a house, it will be understood that in other embodiments, other types of floor plans or other mapping information can be similarly generated, including those for buildings (or other structures or layouts) separate from the house, and including detailed measurements identifying specific rooms or the entire building (or other structure or layout). As another non-exclusive example, although a floor plan of a house or other building can be used to display information to help a viewer navigate the building, in other embodiments, the generated mapping information can be used in other ways. As yet another non-exclusive example, while some embodiments discuss acquiring and using data from one or more types of image acquisition devices (e.g., mobile computing devices and / or separate camera devices), in other embodiments, the one or more devices used may take other forms, such as mobile devices that use some or all of the acquired additional data but do not provide their own computing capabilities (e.g., additional "non-computing" mobile devices), multiple separate mobile devices (whether mobile computing devices or non-computing mobile devices) that each acquire some of the additional data, etc. Additionally, the term "building" herein refers to any partially or completely enclosed structure, typically but not necessarily encompassing one or more rooms that visually or otherwise divide the interior space of the structure. Non-limiting examples of such buildings include houses, apartment buildings or individual apartments therein, condominiums, office buildings, commercial buildings, or other wholesale and retail structures (e.g., shopping malls, department stores, warehouses, etc.). As used herein with reference to the interior of a building, the acquisition location, or other locations (unless the context clearly indicates otherwise), the term “acquisition” or “capture” can refer to any recording, storage, or input of media, sensor data, and / or other information relating to the spatial and / or visual and / or otherwise perceptible characteristics of the interior of a building, or a subset thereof, such as by a recording device or by another device receiving information from a recording device. As used herein, the term “panoramic image” can refer to a visual representation based on, comprising, or divisible into multiple discrete component images originating from substantially similar physical locations in different directions and depicting a wider field of view than any single discrete component image depicts, including images from physical locations with a sufficiently wide field of view to include angles beyond what a person can perceive from a single direction of gaze (e.g., greater than 120°, 150°, or 180°, etc.).As used herein, the term "series" of acquisition locations generally refers to two or more acquisition locations, each accessed at least once in a corresponding sequence, regardless of whether other non-acquisition locations have been accessed in between, and regardless of whether the access to said acquisition locations occurs during a single consecutive time period or at multiple different times, or by a single user and / or device or by multiple different users and / or devices. Additionally, various details are provided in the figures and text for illustrative purposes, but these details are not intended to limit the scope of the invention. For example, the dimensions and relative positions of elements in the figures are not necessarily drawn to scale, and some details are omitted and / or provided more prominently (e.g., via dimensions and positioning) to enhance readability and / or clarity. Furthermore, the same reference numerals may be used in the figures to identify similar elements or actions.

[0023] Figure 1A This is an exemplary block diagram of various devices and systems that can participate in the described technology in some embodiments. Specifically, the target panoramic image 165 is... Figure 1A As shown, it has been captured by one or more mobile computing devices 185 with imaging systems and / or by one or more separate camera devices 186 (e.g., without onboard computing capabilities), such as relative to one or more buildings or other structures, and under the control of an internal capture and analysis (“ICA”) system 160 executed on one or more server computing systems 180 in this example. Figure 1B An example of such panoramic image acquisition location 210 is shown for a specific house 198, as discussed further below, and additional details relating to the automated operation of the ICA system are included herein. Figure 4 Elsewhere. In at least some embodiments, at least some of the ICA systems can be partially executed on the mobile computing device 185 (e.g., as part of the ICA application 154, either supplementing or replacing the ICA system 160 on one or more server computing systems 180) to control the acquisition of target images and optional additional non-visual data, such as those related to the mobile computing device and / or by one or more optional separate camera devices 186 operating in conjunction with the mobile computing device in the vicinity (e.g., within the same room). Figure 1B Further discussion is needed.

[0024] Figure 1AFurther illustrated is a MIGM (Map Information Generation Manager) system 140, which executes on one or more server computing systems 180 to determine the room shape based on a combination of visual data from multiple target images (e.g., panoramic images 165) acquired in each of those rooms, and optionally further generates and provides building floor plans 155 and / or other mapping-related information (e.g., linked panoramic images, etc.) based on the use of the target images and optional associated metadata about their acquisition and linking. Figure 2M Figure 2O (referred to herein as "2-O" for clarity) shows an example of such a floor plan, as discussed further below, and elsewhere in this document include additional details relating to the automated operation of the MIGM system, including information on... Figures 5A to 5C Additional details are provided below. In some implementations, the ICA system 160 and / or the MIGM system 140 may run on the same server computing system, such as if multiple or all of those systems are operated by a single entity or otherwise coordinated with each other (e.g., some or all of the functions in those systems are integrated together into a larger system). In other implementations, the ILDM system may, conversely, operate independently of the ICA system (e.g., without interacting with the ICXA system), such as to obtain target images and / or optionally other information (e.g., other additional images, etc.) from one or more external sources, and optionally store them locally (not shown) along with the MIGM system for further analysis and use.

[0025] In at least some implementations and scenarios, one or more system operator users of the MIGM client computing device 105 can further interact with the MIGM system 140 via network 170, such as to assist some automated operations of the MIGM system for determining room shapes and / or generating floor plans and other mapping information and / or for subsequent use of the determined and generated information in one or more further automated ways. One or more other end users (not shown) of one or more other client computing devices 175 can further interact with the MIGM system 140 and optionally the ICA system 160 via one or more computer networks 170, such as to obtain and use the determined room shape information based on target images, and / or to obtain and optionally interact with the corresponding generated floor plan, and / or to obtain additional information (such as one or more associated target images) and optionally interact with it (e.g., changing between a floor plan view and a view of a specific target image at or near the acquisition location within the floor plan; changing the horizontal and / or vertical viewing direction on which the corresponding view of the panoramic image is displayed, such as determining a portion of the panoramic image pointed to by the current user's viewing direction, etc.). Additionally, although Figure 1ANot explicitly stated, but a floor plan (or a portion thereof) may be linked to or otherwise associated with one or more other types of information, including floor plans of multi-story buildings or other multi-level buildings having multiple associated sub-floor plans of different floors or levels interconnected (e.g., via connecting stairwells), two-dimensional (“2D”) floor plans of a building linked to or otherwise associated with three-dimensional (“3D”) rendered floor plans of the building, etc. In other embodiments, floor plans of multi-story buildings or other multi-level buildings may alternatively include information on all or other floors together and / or may display such information on all or other floors simultaneously. Additionally, although in Figure 1A Not described herein, but in some embodiments, client computing device 175 (or other device, not shown) may additionally receive and use determined room shape information and / or generated floor plan information, such as to control or assist the automated navigation activities of these devices (e.g., autonomous vehicles or other devices), either in lieu of or to supplement the display of the generated information.

[0026] exist Figure 1A In the depicted computing environment, network 170 may be one or more publicly accessible linked networks, such as the Internet, that may be operated by various different parties. In other embodiments, network 170 may have other forms. For example, network 170 may instead be a private network, such as a corporate or university network that is completely or partially inaccessible to non-privileged users. In still other embodiments, network 170 may include both private and public networks, wherein one or more private networks can access one or more public networks and / or access one or more private networks from one or more public networks. Furthermore, in various cases, network 170 may include various types of wired and / or wireless networks. Additionally, client computing device 175 and server computing system 180 may include various hardware components and stored information, as described below. Figure 3 To be discussed in more detail.

[0027] exist Figure 1AIn the example, the ICA system 160 can perform automated operations involving generating multiple target panoramic images (e.g., each a 360-degree panorama around a vertical axis) at multiple associated acquisition locations (e.g., in multiple rooms or other locations within a building or other structure and optionally around some or all of the exterior of the building or other structure), such as for generating and providing a representation of the interior of the building or other structure. In some embodiments, further automated operations of the ICA system may further include: analyzing information to determine the relative position / orientation between each of two or more acquisition locations; creating inter-panorama position / orientation links in the panorama to each of one or more other panoramas based on such determined position / orientation; and then providing information to display or otherwise present the multiple linked panoramic images of the respective acquisition locations within the building, while in other embodiments, some or all of such further automated operations may instead be performed by the MIGM system.

[0028] Figure 1BA block diagram depicts an exemplary building environment in which target panoramic images have been analyzed by a copy of a MIGM system (not shown) to determine the room shapes of one or more or all of the rooms within building 198 (house 198 in this example), and optionally further analyzed by the MIGM system to generate and provide (e.g., present) corresponding building floor plans and / or other mapping-related information (e.g., a set of linked target panoramic images, etc.). Specifically, in the example of 1B, multiple panoramic images are captured at a series of multiple acquisition locations 210 (e.g., inside and outside the house) associated with house 198, such as by a user (not shown) carrying one or more mobile computing devices 185 and / or one or more separate camera devices 186 to capture target images of the multiple acquisition locations 210, along with optional additional non-visual data. Implementations of the ICA system (e.g., ICA system 160 on server computing system 180; some or all copies of the ICA system executing on a user's mobile computing device, such as ICA application system 154 executing in memory 152 on device 185; etc.) can automatically execute or assist in capturing data representing buildings, and in some implementations, further analyze the captured data to generate linked panoramic images, thereby providing a visual representation of the buildings. Although the user's mobile computing device may include various hardware components such as memory 152, display 142, one or more hardware processors 132, one or more image sensors or other imaging systems 135 (e.g., having one or more lenses, associated lights, etc.), optionally one or more depth sensors or other ranging sensors 136, one or more other sensors 148 (e.g., gyroscope 148a, accelerometer 148b, magnetometer or other compass 148c, etc., such as portions of one or more IMUs or inertial measurement units of the mobile device; altimeter; light detector; etc.), optionally a GPS receiver, and optionally other components not shown (e.g., additional non-volatile storage devices; via network 17) 0 and / or the ability to interact with other devices via direct device-to-device communication, such as interacting with an associated camera device 186 or a remote server computing system 180; a microphone; one or more external lights; etc.), in some embodiments, the mobile computing device does not include a ranging sensor 136 or otherwise has access to or uses other specialized equipment to measure the depth of objects in a building relative to the position of the mobile computing device, such that in such embodiments, the relationship between different target panoramic images and their acquisition positions can be determined, in part or entirely, based on the analysis of visual data in different images and / or by using information from other listed hardware components, but without the use of any data from any such ranging sensor 136.Although not described for brevity, one or more camera devices 186 may similarly each include at least one or more image sensors and storage for storing acquired target images, as well as the ability to transmit the acquired target images to other devices (e.g., associated mobile computing device 185, remote server computing system 180, etc.), optionally, and one or more lenses and lights, and optionally in some embodiments, some or all of the other components of the mobile computing device are shown. Additionally, although... Figure 1B Direction indicator 109 is provided for viewer reference and discussion of subsequent examples, but in at least some implementations, mobile computing devices and / or ICA systems may not use this absolute direction information, such as determining the relative direction and distance between panoramic images 210 without considering actual geographic location or direction.

[0029] In operation, the mobile computing device 185 and / or camera device 186 (hereinafter referred to as...) Figure 1B For example, “one or more image acquisition devices” arrive at a first acquisition position 210A in a first room inside the building (in this example, via an entrance passage from the outer door 190-1 to the living room) and capture visual data of a portion of the building interior visible from that acquisition position 210A (e.g., some or all of the first room, and optionally, a small portion of one or more other adjacent or nearby rooms, such as through doorways, hallways, staircases, or other connecting passages from the first room). In at least some cases, the one or more image acquisition devices may be carried or otherwise accompanied by one or more users, while in other implementations and situations, they may be mounted on or carried by one or more self-powered devices that move through the building under their own power. Furthermore, the capture of visual data from the acquisition location can be performed in various ways and through various implementations (e.g., by using one or more lenses that simultaneously capture all image data, by having an associated user rotate his or her body while keeping one or more image acquisition devices stationary relative to the user's body, by rotating one or more image acquisition devices via an automated device on which one or more image acquisition devices are mounted or carried, etc.), and can include recording video at the acquisition location and / or capturing one or more consecutive images at the acquisition location, including capturing visual information depicting multiple objects or other elements (e.g., structural details) visible in images captured from or near the acquisition location (e.g., video frames). Figure 1BIn the example, such objects or other elements include various elements (or structural "wall elements") that are structurally part of the walls of a room in a house, such as doorways 190 and 197 and their doors (e.g., with revolving and / or sliding doors), windows 196, and wall boundaries (e.g., corners or edges) 195 (including corner 195-1 at the northwest corner of house 198, corner 195-2 at the northeast corner of the first room (living room), and corner 195-3 at the southwest corner of the first room). Figure 1B In the example, such objects or other elements may further include other elements within the room, such as furniture 191 to 193 (e.g., sofa 191; chair 192; table 193; etc.), pictures or paintings hanging on the wall, or televisions or other objects 194 (e.g., 194-1 and 194-2), lighting fixtures, etc. One or more image acquisition devices may optionally further capture additional data at or near the acquisition position 210A during rotation (e.g., additional visual data using imaging system 135, additional motion data using sensor module 148, optional additional depth data using range sensor 136, etc.), and may further optionally capture other such additional data as one or more image acquisition devices move to and / or from the acquisition position. In some embodiments, the operation of one or more image acquisition devices may be controlled or facilitated by using one or more programs executed on mobile computing device 185 (e.g., via automated instructions to one or more image acquisition devices or another mobile device (not shown), i.e., carrying these devices through a building under its own power; via instructions to associated users in a room, etc.), such as ICA application system 154 and / or optional browser 162, control system 147 to manage I / O (input / output) and / or for communication and / or networking of device 185 (e.g., receiving instructions from its users and presenting information to users), etc. Users may also optionally provide textual or auditory identifiers associated with acquisition locations, such as “entrance” for acquisition location 210A or “living room” for acquisition location 210B, while in other embodiments, the ICA system may automatically generate such identifiers (e.g., by performing corresponding automated determination through automatic analysis of video and / or other recorded information of the building, such as by using machine learning), or may not use identifiers.

[0030] After sufficient capture has been achieved at the first acquisition location 210A, one or more image acquisition devices (and a user, if present) may proceed to the next acquisition location (such as acquisition location 210B along the travel path 115), whereby the one or more image acquisition devices may optionally record movement data during the movement between acquisition locations, such as visual data and / or other non-visual data from hardware components (e.g., from one or more IMUs 148, from the imaging system 135, from the ranging sensor 136, etc.). At the next acquisition location, the one or more image acquisition devices may similarly capture one or more target images from that acquisition location, and optionally acquire additional data at or near that acquisition location. This process may be repeated from some or all of the rooms of a building, and optionally outside the building, as illustrated for acquisition locations 210C through 210S. Further analysis of the video and / or other images acquired by the image acquisition devices for each acquisition location is performed to generate a target panoramic image for each of acquisition locations 210A through 210S, such as stitching together multiple component images in some embodiments to create a panoramic image and / or matching objects and other elements in different images.

[0031] In addition to generating such panoramic images, in at least some embodiments, further analysis may be performed by the MIGM system (e.g., concurrently with or after image capture) to determine the room shape of each room (and optionally other defined areas, such as decks or other patios or other externally defined areas outside the building), including optionally determining the acquisition location information of each target image, and optionally further determining the building's floor plans and / or other relevant mapping information of the building (e.g., a set of interconnected linked panoramic images, etc.), for example, to "link" at least some of the panoramic images and their acquisition locations together (where exemplary acquisition location 2 is shown for illustration). The copy of the MIGM system can determine the relative position information between pairs of acquisition positions that are visible to each other, store the corresponding panorama links (e.g., links 215-AB, 215-BC, and 215-AC between acquisition positions 210A and 210B, 210B and 210C, and 210A and 210C, respectively), and in some implementations and situations, further link at least one acquisition position that is not visible to each other (e.g., link 215-BE (not shown) between acquisition positions 210B and 210E; link 215-CS (not shown) between acquisition positions 210C and 210S, etc.).

[0032] about Figures 1A to 1BVarious details are provided, but it should be understood that the details provided are non-exclusive examples included for illustrative purposes, and other implementations may be carried out in other ways without some or all of these details.

[0033] Figures 2A to 2V This demonstrates the automatic analysis of visual data from combinations of multiple images captured in a building's rooms, such as based on... Figure 1B Examples of capturing target images within building 198 discussed herein to determine the shape of rooms, and for use in one or more ways, such as within a floor plan of a building partially based on the determined room shape of the rooms, including in some implementations and situations, to further determine and present information about the floor plan of the building.

[0034] In particular, Figure 2A An exemplary image 250a is shown, such as that captured by one or more image acquisition devices. Figure 1B A non-panoramic perspective image (or a northeast-facing subset view of a 360-degree panoramic image taken from acquisition location 210B and formatted in a linear fashion) of the living room of house 198 is shown in this example, further illustrated by a direction indicator 109a to indicate the northeast direction of the image. In the illustrated example, the image includes several visible elements (e.g., lighting fixtures 130a), furniture (e.g., chairs 192-1), two windows 196-1, and a picture 194-1 hanging on the north wall of the living room. Access to the living room (e.g., doorways or other wall openings) is not visible in this image. However, multiple room boundaries are visible in image 250a, including the horizontal boundary between the visible portion of the north wall of the living room and the living room ceiling and floor, the horizontal boundary between the visible portion of the east wall of the living room and the living room ceiling and floor, and the vertical boundary 195-2 between the north and east walls.

[0035] Figure 2B continue Figure 2A Examples are shown, and the image acquisition device is described as being used in... Figure 1B An additional perspective image 250b, captured from acquisition position 210B in the northwest direction in the living room of house 198, further shows a direction indicator 109b to indicate the northwest direction of the captured image. In this exemplary image, a small portion of one of the windows 196-1, along with a portion of window 196-2 and the new lighting fixture 130b, remains visible. Additionally, the horizontal and vertical room boundaries are aligned with... Figure 2A A similar approach can be seen in image 250b.

[0036] Figure 2C continue Figures 2A to 2BExamples are shown, and the image acquisition device is described as being used in... Figure 1B In the living room of house 198, a third perspective image 250c, captured from acquisition position 210B in a southwest direction, further shows a direction indicator 109c to indicate the southwest direction of the image. In this example image, a portion of windows 196-2 remains visible, as does sofa 191, and the visual horizontal and vertical room boundaries are aligned with... Figure 2A and Figure 2B Similar methods can also be seen. This exemplary image further illustrates a wall opening passageway for entering and exiting the living room; in this example, the wall opening passageway is a doorway 190-1 for entering and exiting the living room. Figure 1B (Identify it as a door leading to the outside of the house). It will be understood that various other perspective images can be captured from acquisition location 210B and / or other acquisition locations and displayed in a similar manner.

[0037] Figure 2D It shows Figure 1B Additional information about part of house 198, 255d, including the living room and a limited portion of other rooms to the east of the living room. (See also: Regarding...) Figure 1B and Figures 2A to 2C As discussed, in some embodiments, target panoramic images can be captured at various locations within the house, such as locations 210A and 210B in the living room, and the corresponding visual content of one or both of these obtained target panoramic images is then used to determine the room shape of the living room. Additionally, in at least some embodiments, supplementary images can be captured, such as video or one or more other series of continuous or near-continuous images as one or more image acquisition devices (not shown) move through the interior of the house. In this example, [details omitted]. Figure 1B The information shown is a portion of path 115, and specifically illustrates a series of locations 215 along a path that can capture (e.g., if capturing video data) one or more video frame images (or other series of consecutive or nearly consecutive images) of the interior surrounding a house as one or more image acquisition devices move. Examples of such locations include capture locations 240a to 240c, where other information is related to... Figures 2E to 2J The locations shown relate to video frame images captured at those locations. In this example, location 215 along the path is shown as separated by short distances (e.g., one foot, one inch, a fraction of an inch, etc.), but it will be understood that video captures can be substantially continuous, and therefore, in at least some implementations, only a subset of such captured video frame images (or other images from a series of continuous or nearly continuous images) can be selected and used for further analysis, such as separating images by defined distances and / or separating them by defined time amounts (e.g., one second, a fraction of a second, several seconds, etc.) and / or images based on other criteria.

[0038] Figures 2E to 2J continue Figures 2A to 2D Examples are given, and additional information is shown regarding the analysis of 360° image frames from video captured along path 155, focusing on the living room and, consequently, the estimation of a type of possible room shape. Although not illustrated in these figures, similar techniques can be performed on panoramic images of the target captured by the camera device at two or more acquisition locations 210A, 210B, and 210C, for either supplementary analysis or... Figure 2D The additional image frames shown (e.g., to generate additional estimates of the possible shapes of the room using visual data from the target image) are used instead of analysis. Figure 2D The additional image frame shown. In particular, Figure 2E Includes information 255e, which states that the 360° image frame taken from position 240b will share information about various visible 2D features with the 360° image frame taken from position 240a, but for simplicity, only... Figure 2E This describes a finite subset of this characteristic of a part of the living room. Figure 2E The diagram illustrates exemplary line-of-sight 228 from position 240b to various example features in the room, and similar exemplary line-of-sight 227 from position 240a to the corresponding feature, illustrating the degree of difference between viewpoints at significantly separated capture positions. Therefore, analysis using SLAM and / or MVS and / or SfM techniques corresponding to... Figure 2D A series of images at location 215 can provide a variety of initial information about the characteristics of the living room, such as... Figures 2F to 2I Further explanation.

[0039] In particular, Figure 2F Information 255f shows the northeastern portion of the living room visible in a subset of 360° image frames taken from positions 240a and 240b, and Figure 2GInformation 255g of the northwest portion of the living room, visible in other subsets of 360° image frames taken from positions 240a and 240b, is shown, where various example features of those portions of the living room are visible in both 360° image frames (e.g., corners 195-1 and 195-2, windows 196-1 and 196-2, etc.). As part of the automated analysis of the 360° image frames using SLAM and / or MVS and / or SfM techniques, partial information of planes 286e and 286f corresponding to the portion of the north wall of the living room can be determined based on the detected features, and similarly, partial information 287e and 285f regarding the portions of the east and west walls of the living room can be determined based on corresponding features identified in the images. In addition to identifying this partial planar information of detected features (e.g., each point in the sparse 3D point cloud determined from image analysis), SLAM and / or MVS and / or SfM techniques can also determine information about: the possible positions and orientations / directions 220 of the image subset from capture location 240a and the possible positions and orientations / positions 222 of the image subset from capture location 240b (e.g., the positions of capture locations 240a and 240b respectively). Figure 2F The positions in the middle are 220g and 222g, and optional Figure 2F The image subsets shown are oriented at directions 220e and 222e; and the capture positions are 240a and 240b, respectively. Figure 2G The corresponding positions in the text are 220g and 222g, and optionally... Figure 2G The image subsets shown are oriented in directions 220f and 222f. Although Figure 2F and Figure 2G Only features of a portion of the living room are shown, but it will be understood that other portions of the 360° image frames corresponding to other parts of the living room can be analyzed in a similar manner to determine possible information about the various walls of the room and other features of the living room (not shown). Furthermore, similar analysis can be performed between some or all other images at the selected location 215 in the living room, resulting in multiple determined feature planes from various image analyses that could correspond to portions of the room's walls.

[0040] Figure 2H continue Figures 2A to 2GExamples are given, and information 255h regarding various defined feature planes that could correspond to portions of the west and north walls of the living room is shown from the analysis of 360° image frames captured at positions 240a and 240b. The plane information shown includes defined planes 286g near or at the north wall (and therefore the corresponding possible locations of portions of the north wall), and defined planes 285g near or at the west wall (and therefore the corresponding possible locations of portions of the west wall). As would be expected, there are many variations in the different defined planes of the north and west walls with different features detected in the analysis of the two 360° image frames, such as differences in position, angle, and / or length, as well as missing data for certain portions of the walls, resulting in uncertainty about the actual precise location and angle of each wall. Although... Figure 2H Not shown, but it will be understood that similarly defined feature planes of other living room walls will be detected, as well as defined feature planes corresponding to features not along the walls (e.g., furniture).

[0041] Figure 2I continue Figures 2A to 2H Examples are provided, and information 255i is shown regarding additional determined feature plane information corresponding to portions of the west and north walls of the living room, derived from analysis of various additional 360° image frames selected from additional locations 215 along path 115 in the living room. As would be expected in this example, analysis of other examples provides even greater variations in the different determined planes of the north and west walls. Figure 2I Additional identifying information, such as 295a and 295b, is also shown to aggregate information about various defined feature plane portions in order to identify possible partial locations of the west and north walls. Figure 2J Information 255j explains this. Specifically, Figure 2I Information 291a regarding the orthogonal directions of normals for some of the determined feature planes corresponding to the west wall is shown, along with additional information 288a regarding those determined feature planes. In an exemplary embodiment, the determined feature planes are clustered to represent the hypothetical wall locations of the west wall, and information about the hypothetical wall locations is combined to determine possible wall locations 295a, such as weighting information from various clusters and / or the basic determined feature planes. In at least some embodiments, hypothetical wall locations and / or normal information are analyzed using machine learning techniques to optionally further apply assumptions or other constraints (such as 90° angles, etc.) as part of the machine learning analysis. Figure 2HAs described in information 289, and / or with flat walls) or by applying the results of the analysis, the possible wall locations can be determined. A similar analysis can be performed for the north wall using information 288b about the corresponding determined feature planes and additional information 291b about the obtained orthogonal directions of the normals of at least some of those determined feature planes. Figure 2J The possible partial wall locations 295a and 295b are shown for the west and north walls of the living room, respectively.

[0042] although Figure 2I Not explicitly stated, but understood, similarly determined characteristic planes and corresponding standard orientations of other walls in the living room will be detected and analyzed to determine their possible locations, resulting in an estimated partial overall room shape of the living room based on visual data acquired by one or more image acquisition devices. Furthermore, a similar analysis will be performed for each room in the building, providing an estimated partial room shape for each room. Additionally, although... Figures 2D to 2J Not described herein, but in some embodiments, the analysis of visual data captured by one or more image acquisition devices can be supplemented and / or replaced by the analysis of depth data (not shown) captured in the living room by one or more image acquisition devices, such as generating an estimated 3D point cloud directly from depth data representing the walls and optionally the ceiling and / or floor of the living room. Although Figures 2D to 2J Not explicitly stated herein, but in at least some implementations, other room shape estimation operations can be performed using only a single target panoramic image, such as analysis of visual data of the target panoramic image via one or more trained neural networks, as described in more detail elsewhere herein.

[0043] Figure 2K continue Figures 2A to 2J Examples are provided, and information 255k regarding additional information that can be generated from one or more images of a room and used in one or more ways in at least some embodiments is shown. Specifically, images (e.g., video frames) captured in the living room of house 198 can be analyzed to determine the estimated 3D shape of the living room, such as from a 3D point cloud of features detected in the video frames (e.g., using SLAM and / or SfM and / or MVS techniques, and optionally further based on IMU data captured by one or more image acquisition devices). In this example, information 255k reflects an example portion of such a point cloud of the living room, such as in this example, to... Figure 2CImage 250c corresponds similarly to the northwest direction of the living room (e.g., including the northwest corner 195-1 of the living room and window 196-1). This point cloud can be further analyzed to detect features such as windows, doorways, and openings between rooms; in this example, region 299 corresponding to window 196-1 and boundary 298 corresponding to the north wall of the living room are identified. It will be understood that in other embodiments, this estimated 3D shape of the living room can be determined by using depth data captured in the living room by one or more image acquisition devices, either supplementing or replacing visual data from one or more images captured in the living room by one or more image acquisition devices. Additionally, it will be understood that various other walls and features can be similarly identified in the living room and other rooms of the house 198.

[0044] Figure 2L Additional information 255l is shown, which corresponds to the final estimated room shape of the rooms of house 198 (e.g., 2D room shape 236 of the living room) after determining the final estimated room shape of the rooms relative to each other, in this example, based at least in part on the inter-room passageway connecting the rooms and the matching room shape information of adjacent rooms. In at least some embodiments, this information can be considered as constraints on the positioning of the rooms, and the optimal or otherwise preferred solution for those constraints is determined. Figure 2L Examples of such constraints include allowing connection information between adjacent rooms (e.g., in relation to...). Figures 2E to 2J and / or Figures 2P to 2V In the automated image analysis discussed, the detected channels are matched 231 such that the positions of those channels are co-located, and the shapes of adjacent rooms are matched 232 to connect those shapes (e.g., as shown for rooms 229d and 229e and rooms 229a and 229b). In other embodiments, various other types of information, such as the precise or approximate dimensions of the overall size of the house, can be used to supplement or replace channel-based constraints and / or room shape-based constraints, for room shape location (e.g., based on available additional metadata about the building, analysis of images from one or more image acquisition locations outside the building, etc.). Building exterior information 233 can be further identified and used as constraints (e.g., automated identification based at least in part on channels and other features corresponding to the building exterior, such as windows), for example, to prevent another room from being placed in a location already identified as outside the building. Figure 2L In the example, the final estimated room shape used can be a 2D room shape, or conversely, a 2D version of the 3D final estimated room shape can be generated and used (e.g., by taking horizontal slices of the 3D room shape).

[0045] Figures 2M to 2-O continue Figures 2A to 2L Examples, and show that can be obtained from Figures 2A to 2L and Figures 2P to 2V The mapping information generated by the analysis types discussed in the text. Specifically, Figure 2M An exemplary floor plan 230m is shown, which can be constructed based on the location of a determined final estimated room shape. In this example, the floor plan includes indications of walls, doorways, and windows. In some implementations, such a floor plan may show other information, such as features automatically detected by the analysis operation and / or subsequently added by one or more users. For example, Figure 2NA modified floor plan 230n is shown, including various types of additional information, such as information that can be automatically identified and added to the floor plan 230m from the analysis of visual data from images and / or from depth data, including one or more of the following types of information: room labels (e.g., “living room” for a living room), room dimensions, visual indications of furniture or appliances or other fixed features, visual indications of locations of additional types of associated and linked information (e.g., panoramic and / or perspective images acquired at a specified acquisition location that the end user can choose to further display; audio annotations and / or sound recordings that the end user can choose to further present, etc.), visual indicators of doorways and windows. In other embodiments and situations, some or all of this type of information may instead be provided by one or more MIGM system operator users and / or ICA system operator users. Additionally, when displaying floor plans 230m and / or 230n to the end user, one or more user-selectable controls can be added to provide interactive functionality as part of the GUI (Graphical User Interface) screen 255n, such as indicating the current floor to be displayed, allowing the end user to select different floors to be displayed, etc. In this example, the corresponding example user-selectable control 228 is added to the GUI. Furthermore, in some embodiments, floors or other buildings can be changed directly from the displayed floor plan, such as by selecting the corresponding connecting passage (e.g., stairs leading to different floors), and other visual changes can be made directly from the displayed floor plan by selecting the corresponding displayed user-selectable control (e.g., selecting a control corresponding to a specific image at a specific location, and receiving the display of that image, either replacing or supplementing the previous display of the floor plan from which the image was selected). In other embodiments, information about some or all different floors can be displayed simultaneously, such as in the form of separate sub-floor plans for individual floors or by integrating all room and floor connection information into a single floor plan that is displayed simultaneously. It will be understood that in some implementations various other types of information may be added, in some implementations some of the information of the described types may not be provided, and in other implementations visual indicators of links and associated information and user selections thereto may be displayed and selected in other ways.

[0046] Figure 2-O continue Figures 2A to 2N Examples are provided, and examples are shown that can be disclosed and displayed in this article (e.g., in similar formats). Figure 2NThe additional information 265o generated by automated analysis techniques (within the GUI) is, in this example, a 2.5D or 3D model floor plan of the house. This model 265o can be additional mapping-related information generated based on the floor plan 230m and / or 230n, showing additional information about height to illustrate the visual location of features such as windows and doors within the walls, or conversely, the room shape through a final estimated 3D shape derived from the combination of these elements. Although... Figure 2-O Not described herein, but in some implementations, additional information may be added to the displayed walls, such as images taken during video capture (e.g., rendering and illustrating actual paintings, wallpaper, or other surfaces from the house on the rendered model 265), and / or may be used in other ways to add specified colors, textures, or other visual information to the walls and / or other surfaces.

[0047] Figures 2P to 2V continue Figures 2A to 2-O Examples, where Figure 2P Further information 255p is shown, illustrating an exemplary flow of information processing during automated operation of the MIGM system in at least some embodiments, including a portion 292 that is repeatedly executed as part of iterative refinement of alignment of visual data in multiple target images. Specifically, in Figure 2P In the example, two panoramic images are captured in a specific room (not shown), such as in Figure 1B The example house 198 has two panoramic images captured at locations 210D and 210F in bedroom 1. In this example, both panoramic images are in an isorectangular format and are... Figure 2P In the example, these are labeled as panoramic image 1 241a and panoramic image 2 241b. In this example, the automated operation of the MIGM system also uses the initial estimated pose information of each panoramic image as input, including initial pose information 242a of panoramic image 1 241a and initial pose information 242b of panoramic image 2 241b. Optionally, the height information of the image acquisition device when capturing each of the two panoramic images is also provided as input, including height information 243a of panoramic image 1 241a and height information 243b of panoramic image 2 241b. This height information can be expressed in various ways, such as the relative distance between the floor and the ceiling, the absolute distance between the floor and / or the ceiling, etc. In this example, panoramic image 1 241a is selected as the first image / reference image, and panoramic image 2 241b subsequently adjusts its initial pose information to reflect the pose information of panoramic image 1 241a as part of iterative analysis 292.

[0048] In particular, Figure 2PThe MIGM system 140 receives information 241, 242, and optionally 243 as input and performs processing 281a to analyze panoramic image 1 information (information 241a, 242a, and optionally 243a), including using initial pose information 242a to project two linear perspective images in this example, each image comprising a subset of visual data of panoramic image 1, including linear floor image 1 244a and linear ceiling image 1 245a, and further optionally scaling and / or rotating the visual data in one or both of the two linear images such that corresponding elements in the two linear images have the same size (e.g., reducing the size of the closer surface to correspond to the size of the parent surface when camera height information indicates that the image acquisition device is closer to the floor or ceiling) and the same degree of rotation to aid subsequent comparison and difference analysis. If height information 243a is not available as input for scaling the linear images, in some embodiments, automated operations may further obtain the corresponding information in other ways for scaling activities, such as via analysis of the visual data of panoramic image 1 and / or at least in part based on input from one or more users (not shown). Processing of the information in the panoramic image 1 also includes performing process 282a to analyze the panoramic image 1 and the straight floor and ceiling images 1 to identify structural elements (e.g., 2D elements) of the walls, floor, and ceiling, and to determine the locations of those identified elements in the visual data of those images, such as the estimated partial room shape of the room determined in process 282a based on the information in the panoramic image 1, thereby generating identification element information 246a1 for the panoramic image 1, identification element information 246a2 for the floor image 1, and identification element information 246a3 for the ceiling image 1, as well as the estimated partial room shape 1 248a. Process 282a may include, for example, using one or more trained neural networks (not shown) to identify elements and / or determine their locations and / or estimate partial room shapes, as discussed in more detail elsewhere herein. Similar to the processing 281a and 282a of information for panoramic image 1, the automated operation of the MIGM system further performs processing 281b and 282b of information for panoramic image 2 to generate identification element information 246b1 for panoramic image 2, identification element information 246b2 for straight floor image 2 244b and identification element information 246b3 for straight ceiling image 2 245b, and an estimated partial room shape 2 248b. In this example, initial pose information 242b is used to project the two linear perspective images to each include a subset of the visual data of panoramic image 2, and further optionally scales and / or rotates the visual data in one or both of the two linear images such that corresponding elements in the two linear images have the same size and at the same rotation angle.

[0049] After generating various identification element information 246, the automated operation of the MIGM system further performs processing 283 to analyze the differences between information 246 generated from visual data of two panoramic images (including linear perspective images generated from these panoramic images to contain a subset of their visual data). In at least some embodiments, the difference analysis includes identifying differences in the identified element position information, such as determining offsets in the projected positions of elements. Furthermore, in at least some embodiments, the difference analysis includes determining layout differences in corresponding areas of a room, such as loss measurements of alignment differences in the floor and / or ceiling layouts, either supplementing or replacing the determination of identified element position differences. Processing 283 may include, for example, using one or more trained convolutional neural networks (not shown), as discussed in more detail elsewhere herein. Following the difference analysis, the automated operation continues to determine whether the difference information is below a defined threshold or meets one or more defined criteria. If not, updated pose information 242b is generated for use in the next iteration of the analysis 292 for panoramic image 2, such as using relative difference information 247 relative to the pose information 242a of panoramic image 1 and / or using information 247 regarding alignment loss information from floor / ceiling alignment. In at least some embodiments, updated pose information for panoramic image 2 can be generated at least in part based on changes or increments in three degrees of freedom (e.g., position or orientation) or six degrees of freedom (position and orientation) between the identified element positions from the visual data of panoramic image 1 and the identified element positions from the visual data of panoramic image 2, such as if the difference analysis is at least in part based on the identified element positions. Additionally, in at least some embodiments, if the difference analysis is at least partially based on floor / ceiling alignment loss (whether supplementing or replacing the identified element positional differences), updated pose information can be generated at least partially based on the alignment loss information, such as modifying the layout of corresponding regions (e.g., floor and / or ceiling) in one or more images 2 (e.g., floor image 2 and / or ceiling image 2) to reflect those regions in image 1 (e.g., floor image 1 and / or ceiling image 1), and updating or otherwise adjusting pose information 242b to produce the modified layout of the corresponding regions. After the updated pose information 242b is available, the next iteration begins analyzing information from at least panoramic image 2, such as by performing processing 281b to preserve new straight-line floor and ceiling images 244b and 245b using the updated pose information 242b, and then continuing processing 282b and 283 using those new straight-line images, the process continuing until the pose information of panoramic image 2 is finally updated to produce difference information below a defined threshold. The result of the updated pose information can be considered as the center of the newly projected floor and ceiling images.

[0050] Once the discrepancy information from analysis 283 is determined to be below a predetermined threshold, the automated operation of the MIGM system instead includes continuing to execute process 284 to analyze the combined information from the visual data of both panoramic images 1 and 2 in order to generate a final room shape 249 for the room in which the two panoramic images were acquired, such as a fully enclosed three-dimensional shape with planar surfaces to represent each room wall and the room floor and ceiling (e.g., where each wall has one or more planar surfaces, and where each of the floor and ceiling has one or more planar surfaces). As part of process 284, the outputs of the most recent iteration from process 282b (e.g., the last of multiple iterations) and the most recent iteration from process 282a (e.g., the first iteration if process 282a is not repeated during additional iterations) can be obtained and used, such as identified element location information 246, and optional partial room shapes 248a and 248b estimated from the visual data of panoramic images 1 and 2, respectively. In some implementations and situations, the process 284 may be supplemented or replaced by inputs provided by one or more users, such as one or more users mixing or otherwise combining the estimated partial room shapes 248a and 248b to generate the final room shape 249.

[0051] Although Figure 2P The exemplary processing is shown only for a single room and uses only two panoramic images captured in that room. However, it should be understood that similar processing can be performed on multiple rooms (each room of a multi-room building) and can be performed on more than two panoramic images, such as for N panoramic images, where each of the panoramic images 2-N has a corresponding processing 281b to 281N and a processing 282b to 282N being performed, and where difference analysis and pose information updates are performed individually for each of the panoramic images 2-N until some or all of the panoramic images 2-N have updated pose information that produces difference information between the panoramic image and a first panoramic image / reference panoramic image 1 below a defined threshold, at which point information from the analysis of visual data from all these panoramic images is combined and used in processing 284 to generate the final room shape 249 of the room.

[0052] Figures 2Q to 2S continue Figure 2P Examples, where Figure 2Q The first acquisition location in bedroom 1 is shown (e.g., Figure 1B The target panoramic image 250q (e.g., acquired from a location 210D) is obtained. Figure 2P An example of a panoramic image 1), and in that example it is as follows Figure 2QThe image shown is a 360° panoramic image using a spherical format to simultaneously display all visual content of the target panoramic image. As shown, the visual data of the target panoramic image 250q includes visual representations of the doorway 190-3, ceiling lighting 130q, windows 196-4, bookcase 199a, carpet 199b, and other parts of bedroom 1. In this example, the east wall of the room includes multiple planar surfaces, with a protrusion in the middle of the east wall extending into the room. Figure 2R and Figure 2S continue Figure 2Q Examples are provided, and exemplary linear perspective images 255r and 255s are shown respectively. Optionally, after scaling and / or rotation, pose information can be used to... Figure 2Q A 250q panoramic image of the target is projected to capture the panoramic image of that target. In this example, Figure 2R Image 255r is an exemplary ceiling image (e.g., corresponding to...). Figure 2P The ceiling image 1245a), and includes visual data showing the ceiling light 130q, as well as the window 196-4, the bookcase 199a, and the doorway 190-3. In a similar manner, Figure 2S Image 255s is an exemplary floor image (e.g., corresponding to...). Figure 2P The floor image 1244a) includes visual data showing the carpet 199b and portions of the window 196-4, bookcase 199a and doorway 190-3.

[0053] Figure 2T continue Figures 2P to 2S Examples are shown, and a second sampling location in bedroom 1 is illustrated (e.g., Figure 1B Additional second exemplary target panoramic image 250t (e.g., acquired from acquisition location 210F) Figure 2P Panoramic image 2), and in this example it is as follows Figure 2T The image shown uses a spherical format to simultaneously display all visual content of the target panoramic image. As shown, the visual data of the second target panoramic image 250t shows similar information to the first target panoramic image 250q, but the different acquisition position of the second target panoramic image 250t results in some differences relative to the first target panoramic image 250q (e.g., to show more of the southwest corner of the room that is partially obscured by the bookcase 199a in the first target panoramic image 250q). Figure 2T Further illustrated is an additional exemplary linear perspective image 255t, which can be projected from a second target panoramic image 250t using pose information to capture that target panoramic image, optionally after scaling and / or rotation, and in this example, is a second floor image of bedroom 1 (e.g., corresponding to...). Figure 2PThe floor image 2244b), while the second ceiling image projected from the second target panoramic image 250t is not shown in this example. This second ceiling image (e.g., corresponding to...) Figure 2P The ceiling image 2 245b) can be similarly presented in a manner similar to that of... Figures 2U to 2V The discussion methods are generated and used.

[0054] In particular, Figure 2U and Figure 2V continue Figures 2P to 2T Examples are given, and the differences that can be determined between visual data of two floor images generated from two different panoramic images of two targets captured in the same room are illustrated. In particular, Figure 2U Includes display Figure 2S Information 255u from floor image 255s, where... Figure 2T Additional information about the floor image 255t is overlaid with dashed lines on the floor image 255s to illustrate non-exclusive examples of alignment differences corresponding to the positions of different identified elements. For example, in exemplary information 255u, carpet 199b may have positional offsets 234a for one or more portions of the road in the visual data of the two projected floor images, such as corresponding to the west and east sides of the carpet in this example, and optionally reflecting that the position and / or shape of the carpet in one projected straight floor image differs from its position and / or shape in another projected straight floor image. Similarly, while the southwest corner of the floor in the two projected straight floor images may be aligned 235, the northeast and northwest corners of the floor in the two projected straight floor images may differ by offset 234b, such as to reflect the difference in the position of those corners and / or the shape of the floor in the two floor images. Similarly, for boundaries between walls or for other identified elements, other differences in the position of identified elements may appear, such as offset information 234c shown for the boundary of a portion of the wall that is part of the east wall of bedroom 1. It should be understood that similar positional difference information can be determined for all elements identified in two straight floor images, and similar positional difference information can be determined in the visual data of two straight ceiling images projected from two different target panoramic images and / or in the visual data of the two different target panoramic images themselves. Furthermore, in embodiments and cases where loss information is determined between the layout of areas in a room (such as the floor and / or ceiling), other differences in the floor layout between the two projected straight floor images, such as offset information for offset 234b, can be determined. The floor layout in the second floor image is then modified to reduce or eliminate the difference from the floor layout in the first floor image, and the modified layout in the second floor image is used to determine newly adjusted second pose information for use with the second target panoramic image in the next analysis iteration.

[0055] Figure 2V Including similar Figure 2U Information 255v, and again shown Figure 2T The floor image is 255s, but Figure 2V Additional information shown using dashed lines comes from an updated second straight-line floor image (not shown), which is relative to... Figure 2U The difference analysis discussed follows the projection of updated pose information from the second target panoramic image. In this example, the updated second pose information based on the second target panoramic image is closer to the visual data aligning the two target panoramic images (e.g., when capturing target panoramic images at acquisition positions 210D and 210F, it is closer to reflect the actual difference between the positions of acquisition positions 210D and 210F, and reflects the actual difference between the orientations of the image acquisition devices, if any), such as differences relative to offsets 234a, 234b, and / or 234c. Figure 2V relative to Figure 2U The differences in the location of identified elements and / or alignment losses from the layout of the floor and / or ceiling (and / or other areas) can be reduced or eliminated. It should be understood that multiple iterations can be performed to further reduce or eliminate other differences in the location of identified elements and / or alignment losses from the layout of the floor and / or ceiling (and / or other areas), although exemplary information corresponding to additional iterations is not shown in this example. After the last iteration, the combined alignment visual data of the two target panoramic images can be used to generate the 3D room shape of bedroom 1, as discussed in more detail elsewhere in this document.

[0056] Regarding Figures 2A to 2V Various details are provided, but it will be understood that the details provided are non-exclusive examples included for illustrative purposes, and other implementations may be carried out in other ways without some or all of these details.

[0057] Figure 3This is a block diagram illustrating one or more server computing systems 300 implementing an embodiment of the MIGM system 340, and one or more server computing systems 380 implementing an embodiment of the ICA system 387. These server computing systems and the MIGM system can be implemented using multiple hardware components that form electronic circuitry suitable for and configured to perform at least some of the techniques described herein when operating in conjunction. In the illustrated embodiments, each server computing system 300 includes one or more hardware central processing units (“CPU”) or other hardware processors 305, various input / output (“I / O”) components 310, storage devices 320, and memory 330. The illustrated I / O components include a display 311, a network connection 312, a computer-readable media drive 313, and other I / O devices 315 (e.g., a keyboard, mouse or other pointing device, microphone, speaker, GPS receiver, etc.). Each server computing system 380 may include hardware components similar to those of server computing system 300, including one or more hardware CPU processors 381, various I / O components 382, ​​storage devices 385, and memory 386. However, for the sake of brevity, some details of server 300 are omitted from server 380.

[0058] The server computing system 300 and the executing MIGM system 340 can communicate with other computing systems and devices via one or more networks 399 (e.g., the Internet, one or more cellular telephone networks, etc.), such other computing systems and devices include: a user client computing device 390 (e.g., for viewing floor plans, associated images, and / or other relevant information); an ICA server computing system 380; one or more mobile computing devices 360; optionally one or more camera devices 375; optionally other navigable devices 395 that receive and use floor plans and / or determined room shapes and optionally other generated information for navigation purposes (e.g., for use by semi-autonomous or fully autonomous vehicles or other devices); and optionally other computing systems not shown (e.g., for storing and providing additional information related to buildings; for capturing data inside buildings; for storing and providing information to client computing devices, such as additional supplementary information associated with images and the buildings or other surrounding environment they cover; etc.). In some implementations, some or all of one or more camera devices 375 may communicate directly (e.g., wirelessly and / or via cable or other physical connection, and optionally in a peer-to-peer manner) with one or more associated mobile computing devices 360 in their vicinity (e.g., to send captured target images, receive instructions to initiate target image acquisition, etc.), whether in addition to or instead of performing communication via network 399, and wherein such associated mobile computing devices 360 are capable of providing captured target images and optionally other captured data received from one or more camera devices 375 via network 399 to other computing systems and devices (e.g., server computing systems 380 and / or 300).

[0059] In the illustrated embodiments, the implementation of the MIGM system 340 is executed in memory 330 to perform at least some of the described techniques, such as using processor 305 to execute software instructions of system 340 in a manner that configures processor 305 and computing system 300 to perform automated operations that implement those described techniques. The illustrated embodiments of the MIGM system may include one or more components (not shown) to each perform a portion of the functionality of the MIGM system, and the memory may optionally execute one or more other programs 335. As a specific example, in at least some embodiments, a copy of the ICA system may be executed as one of the other programs 335, such as replacing or supplementing ICA system 387 on server computing system 380. The MIGM system 340 may also store and / or retrieve various types of data on storage device 320 (e.g., in one or more databases or other data structures) during its operation, such as information 321 about the target panoramic image (e.g., acquired by one or more camera devices 375) and associated projected linear perspective images, information 325 about the acquisition pose information determined for the target panoramic image (optionally including initial and updated pose information), information 322 about the identified elements of image 321 and their determined positions (e.g., generated by the MIGM system during its automated operation), information 323 about the final 3D room shape of the room determined from visual data of the target image and optional intermediate estimated partial room shapes (e.g., generated by the MIGM system during its automated operation), and various other types of data. Floor plan information and other building mapping information 326 (e.g., generated and saved 2D floor plans, 2D room shapes and locations with wall elements and other elements on those floor plans, and optional additional information such as building and room dimensions for associated floor plans, existing images with specified locations, annotation information, etc.; generated and saved 2.5D and / or 3D model floor plans similar to 2D floor plans but also including height information and 3D room shapes, etc.); user information 328 about the user of the client computing device 390 and / or the operator user of the mobile device 360 ​​interacting with the MIGM system, training data and / or the resulting trained neural network 327 optionally used with one or more convolutional neural networks, and various other optional types of additional information 329.During its operation, ICA system 387 can similarly store and / or retrieve various types of data on storage device 385 (e.g., in one or more databases or other data structures) and provide some or all of such information to MIGM system 340 for its use (whether in a push and / or pull manner), such as images 393 (e.g., 360° target panoramic images captured by one or more camera devices 375 and sent by those camera devices and / or one or more intermediate associated mobile computing devices 360 to server computing system 380), and optional various types of additional information 397 (e.g., various analytical information related to the presentation or other use of one or more building interiors or other environments captured by the ICA system).

[0060] User client computing device 390 (e.g., a mobile device), mobile computing device 360, camera device 375, other navigable devices 395, and some or all of other computing systems may similarly include some or all of the same type of components described for server computing systems 300 and 380. As a non-limiting example, mobile computing device 360 ​​is each shown including one or more hardware CPUs 361, I / O components 362, storage devices 365, imaging systems 364, IMU hardware sensors 369, optionally a depth sensor 363, and memory 367, wherein one or both of a browser and one or more client applications 368 (e.g., applications specific to the MIGM system and / or ICA system) optionally execute within memory 367, such as participating in communication with the MIGM system 340, ICA system 387, associated camera device 375, and / or other computing systems. Although specific components are not described for other navigable devices 395 or client computing systems 390, it will be understood that they may include similar and / or additional components.

[0061] We will also learn about computing systems 300 and 380 and camera device 375, as well as Figure 3Other systems and devices included herein are merely illustrative and are not intended to limit the scope of the invention. Systems and / or devices may instead each comprise multiple interacting computing systems or devices and may connect to other devices not specifically described, including via Bluetooth communication or other direct communication, through one or more networks (such as the Internet), via the Web, or via one or more dedicated networks (e.g., mobile communication networks, etc.). More generally, devices or other computing systems may include any combination of hardware that, when programmed or otherwise configured with specific software instructions and / or data structures, can interact and perform functions of the described type, including but not limited to desktop computers or other computers (e.g., tablet computers, tablet computers, etc.), database servers, network storage devices and other network devices, smartphones and other cellular phones, consumer electronic devices, wearable devices, digital music player devices, handheld gaming devices, PDAs, wireless phones, internet-connected appliances, camera devices and accessories, and various other consumer products including suitable communication capabilities. Additionally, in some embodiments, the functionality provided by the illustrated MIGM system 340 may be distributed across various components, some of the functions described in the MIGM system 340 may not be provided, and / or other additional functions may be provided.

[0062] It will also be understood that, although various entries are described as being stored in memory or on storage devices during use, these entries, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software components and / or system may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. Thus, in some embodiments, some or all of the described techniques may be executed by hardware components including one or more processors and / or memory and / or storage devices, such as by executing software instructions of said one or more software programs and / or by storing such software instructions and / or data structures, and such as by executing algorithms as described in the flowcharts and other disclosures herein. Furthermore, in some implementations, some or all of the system and / or components may be implemented or provided in other ways, such as by one or more means implemented partially or entirely in firmware and / or hardware (e.g., rather than as means wholly or partially implemented by software instructions configuring a particular CPU or other processor), including but not limited to one or more application-specific integrated circuits (ASICs), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field-programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc. Some or all of the components, system, and data structures may also be stored (e.g., as software instructions or structured data) on non-transitory computer-readable storage media, such as hard disks or flash drives or other non-volatile storage devices, volatile or non-volatile memories (e.g., RAM or flash RAM), network storage devices, or portable media articles (e.g., DVDs, CDs, optical discs, flash memory devices, etc.) for access by an appropriate drive or via an appropriate connection. In some embodiments, the system, components, and data structures can also be transmitted over a variety of computer-readable transmission media (e.g., as part of a carrier wave or other analog or digital propagation signal) via generated data signals, including wireless-based and wired / cable-based media, and can take various forms (e.g., as part of a single or multiple analog signal, or as multiple discrete digital packets or frames). In other embodiments, such a computer program product can also take other forms. Therefore, embodiments of this disclosure can be practiced using other computer system configurations.

[0063] Figure 4 An exemplary flowchart illustrating an implementation of ICA system routine 400 is shown. This routine can be, for example... Figure 1A ICA system 160, Figure 3 The ICA system 387 and / or the ICA system otherwise described herein are used for operations such as acquiring 360° panoramic images of the target and / or other images within the building or other structure (e.g., for subsequently generating associated floor plans and / or other mapping information, such as those generated by implementations of the MIGM system routines, wherein regarding Figures 5A to 5C An example of such a routine is illustrated; used to subsequently determine the acquisition location and, optionally, the acquisition orientation, of the target image. While part of example routine 400 is discussed regarding the acquisition of a specific type of image at a specific location, it will be understood that this or similar routines can be used to acquire video or other data (e.g., audio) and / or other types of non-panoramic images, whether as a substitute for or supplement to such images. Furthermore, although the illustrated implementation acquires and uses information from the interior of the target building, it will be understood that other implementations can perform similar techniques for other types of data, including information for non-building structures and / or for the exterior of one or more target buildings of interest. Moreover, some or all of the routines can be executed on a mobile device used by a user to participate in acquiring image information and / or associated additional data, and / or executed by a system remotely connected to such a mobile device.

[0064] The described implementation of the routine begins at box 405, where an instruction or information is received. At box 410, the routine determines whether the received instruction or information instructs the acquisition of data representing a building (e.g., inside a building), and if not, proceeds to box 490. Otherwise, the routine proceeds to box 412 to receive an instruction (e.g., from a user of a mobile computing device associated with one or more camera devices) to begin an image acquisition process at a first acquisition location. After box 412, the routine proceeds to box 415 to perform image acquisition activities at the acquisition location, acquiring at least one 360° panoramic image (and optionally one or more additional images and / or other additional data, such as from IMU sensors and / or depth sensors) at the acquisition location for the target building of interest, such as to provide at least 360° horizontal coverage around a vertical axis. The routine may also optionally obtain annotations and / or other information from the user regarding the acquisition location and / or surrounding environment, such as information for later use in presenting information about the acquisition location and / or surrounding environment.

[0065] After box 415 is completed, the routine continues to box 425 to determine if there are additional acquisition locations for acquiring images, such as based on corresponding information provided by the user of the mobile computing device and / or meeting specified criteria (e.g., at least two panoramic images to be captured in each of some or all rooms of the target building and / or in each of one or more areas outside the target building). If so, the routine continues to box 427 to optionally initiate the acquisition of linking information (such as visual data, acceleration data from one or more IMU sensors, etc.) during movement of the mobile device along a path away from the current acquisition location toward the next acquisition location of the building. As described elsewhere herein, the captured linking information may include additional sensor data (e.g., from one or more IMUs or inertial measurement units on the mobile computing device or otherwise carried by the user) and / or additional visual information recorded during such movement (e.g., panoramic images, other types of images, panoramic or non-panoramic video, etc.), and in some embodiments, the linking information may be analyzed to determine changes in the mobile computing device's posture (position and orientation) during movement and information about the shape of the enclosed room (or other area) and the path of the mobile computing device during movement. The initiation of capturing such link information can be performed in response to explicit instructions from a user of the mobile computing device or based on one or more automated analyses of information recorded from the mobile computing device. Additionally, in some embodiments, the routine may optionally determine and provide one or more guiding cues regarding the movement of the mobile device, the quality of sensor data, and / or visual information captured during movement to the next acquisition location (e.g., by monitoring the movement of the mobile device), including information about associated lighting / ambient conditions, the desirability of capturing the next acquisition location, and any other suitable aspects of capturing link information. Similarly, the routine may optionally obtain annotations and / or other information from the user regarding the travel path, such as information for later use in presenting images of that travel path or the resulting inter-panel image link. In block 429, the routine then determines that the mobile computing device (and one or more associated camera devices) has reached the next acquisition location (e.g., based on instructions from the user, based on the user's forward movement stopping within at least a predetermined time, etc.) to serve as the new current acquisition location, and returns to block 415 to perform an acquisition location image acquisition activity for that new current acquisition location.

[0066] If instead of determining in box 425 that there is no longer a capture location where image information of the current building or other structure needs to be captured, the routine proceeds to box 430 to optionally analyze the capture location information of the building or other structure, such as identifying possible additional coverage (and / or other information) to be captured inside the building or otherwise associated with the building. For example, the ICA system may provide the user with one or more notifications about the information captured during the capture of multiple capture locations and optionally corresponding linking information, such as if it is determined that one or more segments of the recorded information are of insufficient or undesirable quality or do not appear to provide complete building coverage. Furthermore, in at least some embodiments, if minimum image criteria (e.g., minimum number and / or type of images) have not been met by the captured images (e.g., at least two panoramic images in each room, panoramic images within a maximum specified distance from each other), the ICA system may prompt or instruct the capture of additional panoramic images to meet such criteria. Following box 430, the routine continues to box 435 to optionally preprocess the acquired 360° panoramic image of the target, which is then subsequently used to generate relevant mapping information (e.g., to format the image in a spherical format, to determine the vanishing lines and vanishing points of the image, etc.). In box 480, these images and any associated generated or acquired information are stored for later use.

[0067] If instead it is determined in box 410 that the instructions or other information described in box 405 are not for acquiring images and other data representing a building, then the routine continues to box 490 to perform any other indicated operations as appropriate (such as any housekeeping task), configure parameters for use in various operations of the system (e.g., based at least in part on information specified by the system's user, such as a user for capturing mobile devices inside one or more buildings, an operator user of the ICA system, etc.), obtain and store other information about the system's user, respond to requests for information generated and stored, etc.

[0068] After box 480 or 490, the routine proceeds to box 495 to determine whether to continue, such as until an explicit termination instruction is received, or only if an explicit continue instruction is received. If it is determined to continue, the routine returns to box 405 to wait for additional instructions or information, and if not, it proceeds to step 499 and terminates.

[0069] Figures 5A to 5C An exemplary implementation of a flowchart for a Mapping Information Generation Manager (MIGM) system routine 500 is shown. This routine can be implemented, for example, by executing... Figure 1A MIGM system 140, Figure 3The MIGM system 340 and / or the MIGM system described elsewhere herein are used to perform actions such as determining the room shape (or other defined area) by analyzing and combining information from multiple panoramic images acquired in the room, generating a floor plan of the building or other defined area based at least in part on one or more images of the area captured by a mobile computing device and optional additional data, and / or generating other mapping information of the building or other defined area based at least in part on one or more images of the area and optional additional data captured by a mobile computing device. Figures 5A to 5C In the example, the room shape determined for the room is a 3D fully enclosed combination of planar surfaces to represent the room's walls, ceiling, and floor, and the mapping information generated for the building (e.g., a house) includes 2D floor plans and / or 3D computer model floor plans. However, in other implementations, other types of room shapes and / or mapping information may be generated and used in other ways, including for other types of structures and defined areas, as discussed elsewhere herein.

[0070] The described implementation of the routine begins at box 505, where information or instructions are received. The routine continues to box 510 to determine whether image information is already available for analysis of one or more rooms (e.g., some or all of the indicated building), or whether such image information should be acquired at this time. If it is determined in box 510 that some or all of the image information should be acquired at this time, the routine continues to box 512 to acquire such information, optionally waiting for one or more users or devices to move through one or more rooms of the building and acquire panoramic or other images at one or more acquisition locations in one or more rooms (e.g., multiple acquisition locations in each room of the building), optionally along with metadata information about the acquisition and / or interconnection information related to movement between acquisition locations, as discussed in more detail elsewhere herein. Figure 4 An exemplary implementation of an ICA system routine for performing such image acquisition is provided. If instead it is determined in box 510 that no image is currently being acquired, the routine proceeds to box 515 to acquire existing panoramic or other images from one or more acquisition locations in one or more rooms (e.g., multiple acquisition locations in each room of a building), optionally along with metadata information regarding acquisition and / or interconnection information related to movement between acquisition locations, such as in some cases which may have been provided along with corresponding instructions in box 505.

[0071] Following box 512 or 515, the routine continues to box 520, where it determines whether to generate a set of linked target panoramic images (or other images) of a building or other group of rooms, and if so, continues to box 525. In box 525, the routine selects at least some pairs of images (e.g., based on a pair of images with overlapping visual content), and for each pair, determines the relative orientation between the images of the pair (whether it is a movement directly from the acquisition position of one image of the pair to the acquisition position of the other image of the pair, or conversely, a movement between start and end acquisition positions via one or more other intermediate acquisition positions of other images) based on shared visual content and / or other captured link interconnection information (e.g., movement information) associated with the images of the pair. In box 525, the routine further uses at least the relative orientation information of the image pair to determine the global relative positions of some or all of the images relative to each other in a common coordinate system, such as to create a virtual tour through which an end user can move from any of the images to one or more other images linked to the starting image (e.g., via user-selectable controls displayed for each such other linked image), and similarly move from the next image to one or more additional images linked to the next image, and so on. Additional details regarding the creation of such groups of linked images are included elsewhere in this document.

[0072] Following box 525, or if, conversely, it is determined in box 520 that the instructions or other information received in box 505 are not used to determine the linked image group, the routine continues to box 530 to determine whether the instructions received in box 505 instruct the determination of the shape of one or more rooms from previously or currently acquired images in the rooms (e.g., from multiple panoramic images acquired in each room) without generating additional mapping-related information for the building in which the rooms are located, and if so, continues to box 543. Otherwise, the routine continues to box 535 to determine whether the instructions received in box 505 instruct the generation of floor plans and optional other mapping information for the indicated building, and if so, the routine continues to perform boxes 537 through 585 to do so, and otherwise continues to box 590.

[0073] In box 537, the routine may optionally obtain additional information about the building, such as from activities performed during image acquisition and optional image analysis, and / or from one or more external sources (e.g., online databases, information provided by one or more end users, etc.). This additional information may include, for example, the building’s external dimensions and / or shape, additional images and / or annotations acquired corresponding to specific locations within the building (optionally for locations different from the acquisition locations of the panoramas or other images acquired), additional images and / or annotations acquired corresponding to specific locations outside the building (e.g., around the building and / or for other structures on the same property), etc.

[0074] Following box 537, the routine continues to box 540 to determine whether to generate room shapes for rooms enclosing the acquired building image to generate the building's floor plan, and if not (e.g., if room shape information is already available for rooms in the building), it continues to box 570. Otherwise, if it is determined in box 540 to determine room shapes for generating the building's floor plan, or if one or more room shapes are determined from the acquired image in box 530 without generating other mapping-related information, the routine continues to boxes 543 through 565 to generate room shapes for one or more rooms.

[0075] Specifically, the routine in box 543 continues to select the next room (starting from the first room), in which a spherical panoramic image acquired is available to determine initial pose information for each of those panoramic images (e.g., acquisition metadata of the panoramic image is provided), and optionally, additional metadata for each panoramic image (e.g., acquisition height information of the camera device or other image acquisition device used to acquire panoramic images relative to the floor and / or ceiling). Following box 543, the routine continues to box 545, where, for each panoramic image, the routine projects one or more perspective images, as a subset of the panoramic image, using the current pose information of the panoramic image and optional additional metadata, such as one or both of the floor view and ceiling view of the room from which the panoramic image was acquired. Following box 545, the routine continues to box 547, where, for each panoramic image, the routine analyzes the panoramic image and its projected perspective image to identify the positions of wall and floor and ceiling elements, and to determine an estimated room shape based on the visual data from these images. Analysis of visual data from panoramic images acquired in a room may include identifying structural features of the room's walls (e.g., windows, doorways and staircases, as well as wall openings and connecting passages between other rooms, wall boundaries and / or reception and / or floors, etc.) and determining the location of those identified features within a defined room shape, optionally by generating 3D point clouds of some or all of the room's walls and optionally the ceiling and / or floor (e.g., by analyzing at least the visual data from the panoramic images and additional data optionally acquired by the image acquisition device or associated mobile computing device, such as using one or more of SfM, SLAM, or MVS analysis). Elsewhere in this document, additional details are included regarding determining the room shape and identifying additional information about the room, including initial estimated acquisition pose information for the images acquired in the room.

[0076] Following box 547, the routine continues to box 550, where it determines whether to identify differences between visual data of different panoramic images (including their perspective image subsets) based on differences in the identified element positions or differences in the layout of the floor and / or ceiling alignment (e.g., using distinguishable rendering operations). While the illustrated implementation of the routine uses only one type of difference information at a time, other implementations may use both types of difference information, either simultaneously or sequentially. If it is determined in box 550 that element position information is to be used, the routine continues to box 553, where it analyzes the differences in the identified element positions generated from visual data of a first panoramic image / reference panoramic image (including one or more perspective image subsets for the first panoramic image) and visual data of one or more other second panoramic images (including one or more perspective image subsets for each second panoramic image), such as by using one or more convolutional neural networks. If, conversely, floor / ceiling alignment information is determined to be used in box 550, the routine continues to box 557, where it analyzes the differences in floor and / or ceiling layout generated from visual data (including one or more perspective image subsets of the first panoramic image) and visual data (including one or more perspective image subsets of each second panoramic image) from one or more other second panoramic images, such as by using one or more convolutional neural networks. Following box 553 or 557, the routine continues to box 560, where it determines whether the determined differences (e.g., an aggregation of multiple individual difference types) are below a defined threshold (or otherwise meet one or more defined criteria), and if not, it continues to box 563, where it uses the difference information to update the pose information of one or more other second panoramic images relative to the first panoramic image, and returns to box 545. Otherwise, the routine continues to box 565, where it combines current information generated from the first and second panoramic images (such as identified element locations and optionally estimated room shapes) and analyzes this combined information to generate the final room shape, such as by using one or more convolutional neural networks. Following box 565, the routine continues to box 567, where it determines whether there are more rooms whose room shapes can be determined based on panoramic images taken in those rooms, and if so, returns to box 543.

[0077] If it is determined in box 567 that no more rooms can be generated for room shapes, or if the room shapes are uncertain in box 540, the routine continues to box 570 to determine (e.g., based at least in part on the room shapes determined from boxes 543 to 565, and optionally additional information about how the determined room shapes are positioned relative to each other) whether to further generate floor plans for the building. If not, such as when only room shapes are determined without generating further mapping information for the building, the routine continues to box 588. Otherwise, the routine continues to box 575 to retrieve room shapes (e.g., the room shapes generated in box 565) or otherwise obtain the room shapes of the building's rooms (e.g., based on manually provided input), whether 2D or 3D room shapes, and then continues to box 577. In box 577, the routine uses room shapes to create an initial 2D floor plan, such as by determining the corresponding 2D room shapes using wall location information of the 3D room shapes, by connecting the inter-room passages within their respective rooms, by optionally locating the room shapes around determined acquisition locations of the target image (e.g., if the acquisition location locations are interconnected), and by optionally applying one or more constraints or optimizations. This floor plan may include, for example, the relative positions and shapes of various rooms without providing any actual size information for individual rooms or the building as a whole, and may further include multiple linked or associated submaps of the building (e.g., to reflect different floors, levels, sections, etc.). The routine further associates the positions of doors, wall openings, and other identified wall elements on the floor plan. Following box 577, the routine optionally performs one or more steps 580 through 583 to determine additional information and associate it with the floor plan. In box 580, the routine optionally estimates some or all of the dimensions of the room, such as based on analysis of images and / or their acquired metadata or based on overall dimensional information obtained for the exterior of the building, and correlates the estimated dimensions with the floor plan. It will be understood that, if sufficiently detailed dimensional information is available, architectural drawings, engineering drawings, etc., can be generated from the floor plan. Following box 580, the routine continues to box 583 to optionally correlate further information with the floor plan (e.g., having specific rooms or other locations within the building), such as additional images and / or annotation information for the specified locations. In box 585, if the room shape from box 575 is not a 3D room shape, the routine further estimates the height of some or all of the walls in the room, such as based on the analysis of the image and optionally the size of known objects in the image, as well as information about the camera height when the image was acquired, and uses that height information to generate a 3D room shape for the room. The routine further uses the 3D room shape (whether from box 575 or box 585) to generate a 3D computer model floor plan of the building, where the 2D and 3D floor plans are correlated with each other.

[0078] Following box 585, or if instead an uncertain floor plan is determined in box 570, the routine continues to box 588 to store the determined room shape and / or generated mapping information and / or other generated information, and optionally further uses some or all of the determined and generated information, such as providing the generated 2D floor plan and / or 3D computer model floor plan to be displayed on one or more client devices and / or provided to one or more other devices for automated navigation of those devices and / or associated vehicles or other entities, to similarly provide and use panoramic images of the determined room shape and / or a set of linked views and / or information about additional information determined based on the room contents and / or passageways between rooms, etc.

[0079] If, instead of determining in box 535, that the information or instruction received in box 505 does not generate surveying information for the indicated building, the routine proceeds to box 590 to perform one or more other indicated operations as appropriate. Such other operations may include, for example, receiving and responding to requests for and / or previously determined room shapes and / or other generated information (e.g., requests for such information for use by the ILDM system, requests for displaying such information on one or more client devices, requests for providing such information to one or more other devices for automated navigation, etc.), obtaining and storing information about the building for use in later operations (e.g., information about room dimensions, number or type, total building area, adjacent or nearby buildings, adjacent or nearby vegetation, exterior images, etc.), etc.

[0080] After box 588 or 590, the routine continues to box 595 to determine whether to continue, such as until an explicit termination instruction is received, or only if an explicit continue instruction is received. If it is determined to continue, the routine returns to box 505 to wait for and receive additional instructions or information; otherwise, it continues to box 599 and terminates.

[0081] Although there is no information about Figures 5A to 5CThe automated operations illustrated in the exemplary embodiments are described, but in some embodiments, human users can further assist in facilitating certain operations of the MIGM system, such as operator users and / or end users of the MIGM system providing one or more types of input for further use in subsequent automated operations. As a non-exclusive example, such human users may provide one or more types of input, as follows: providing input to help link a set of images, such as providing input in box 525, which is used as part of the automated operation of that box (e.g., to specify or adjust the initial automatically determined orientation between one or more pairs of images, to specify or adjust the initial automatically determined final global position of some or all images relative to each other, etc.); providing input in box 537, which is used as part of subsequent automated operation, such as information about one or more display types of buildings; providing input about box 543, which is used as part of subsequent automated operation, such as specifying or adjusting the initially automatically determined pose information (whether initial pose information or subsequently updated pose information) for one or more panoramic images, to specify or adjust the initial automatically determined information about the acquisition height information and / or other metadata for one or more panoramic images, etc.; providing input about box 545, which is used as part of subsequent automated operation, such as adjusting one or more initially automatically determined projected perspective images (e.g., the position of specific elements of the images); providing input about box 547, which is used as part of subsequent automated operation. As part of an automation operation, such as specifying or adjusting the initially automatically determined element positions and / or estimated room shapes; providing inputs for boxes 553 and / or 557, which are used as part of subsequent automation operations, such as adjusting information about the initial automatic determination of differences between visual data from different panoramic images; providing inputs for box 563, which is used as part of subsequent automation operations, such as specifying or adjusting information about the initial automatic determination of updated pose information; providing inputs for box 565, which is used as part of subsequent operations, such as manually combining information from multiple estimated room shapes to create a final room shape, to specify or adjust initial automatic determination information about the final room shape, etc.; providing inputs for box 577, which is used as part of subsequent operations, such as specifying or adjusting the initially automatically determined room shape positions within the generated floor plan and / or specifying or adjusting the initially automatically determined room shapes themselves within such floor plan; providing inputs for one or more of boxes 580 and 583 and 585, which are used as part of subsequent operations, such as specifying or adjusting one or more types of initial automatic determination information discussed with respect to these boxes; and so on. Additional details regarding the implementation of an approach in which one or more human users provide input for additional automated operations of the MIGM system are included elsewhere in this document.

[0082] Figure 6 An exemplary implementation of a flowchart for a building map viewer system routine 600 is shown. This routine can be implemented, for example, by executing... Figure 1A Map viewer client computing device 175 and its software system (not shown), Figure 3 The client computing device 390 and / or mobile computing device 360, and / or a mapping information viewer or presentation system as described elsewhere herein, may perform actions such as receiving and displaying additional mapping information (e.g., 2D or 3D floor plans, etc.) that defines the shape of a room and / or a defined area, optionally including visual indications of one or more determined image acquisition locations; and optionally displaying additional information (e.g., images) associated with a particular location within the mapping information. Figure 6 In the example presented, the mapping information is for the interior of a building (such as a house), but in other implementations, other types of mapping information may be presented for other types of buildings or environments and used in other ways, as discussed elsewhere in this document.

[0083] The described implementation of the routine begins at box 605, where an instruction or message is received. At box 610, the routine determines whether the received instruction or message instructs the display or otherwise presentation of information representing the interior of the building, and if not, proceeds to box 690. Otherwise, the routine proceeds to box 612 to retrieve instructions on the shapes of one or more rooms of the building or floor plans or other generated mapping information of the building, and optionally associated linked information on the surrounding locations inside and / or outside the building, and selects an initial view of the retrieved information (e.g., a view of the floor plan, a particular room shape, etc.). In box 615, the routine then displays or otherwise presents the current view of the retrieved information and waits for user selection in box 617. Following the user selection in box 617, if it is determined in box 620 that the user selection corresponds to adjusting the current view of the current location (e.g., changing one or more aspects of the current view), the routine continues to box 622 to update the current view according to the user selection, and then returns to box 615 to update the displayed or otherwise presented information accordingly. The user selection and corresponding update of the current view may include, for example, displaying or otherwise presenting an associated link of information (e.g., a specific image associated with a displayed visual indication of the determined acquisition location, such as overlaying the associated link information on at least some of the previous display), and / or changing the way the current view is displayed (e.g., zooming in or out; rotating information where appropriate; selecting a new portion of the floor plan to display or otherwise present, such as where some or all of the new portion was previously invisible, or where the new portion is a subset of previously visible information, etc.).

[0084] If instead it is determined in box 610 that the instructions or other information received in box 605 will not present information representing the interior of the building, the routine continues to box 690 to perform other indicated operations as appropriate (such as any housekeeping task), configure parameters for use in various operations of the system (e.g., at least in part based on information specified by the system user, such as a user for capturing one or more mobile devices inside the building, an operator user of the MIGM system, etc., including personalized information display for a particular user based on his / her preferences), to obtain and store other information about the system user, respond to requests for the generated and stored information, etc.

[0085] After box 690, or if it is determined in box 620 that the user selection does not correspond to the current building area, the routine proceeds to box 695 to determine whether to continue, such as until an explicit termination instruction is received, or only if an explicit continue instruction is received. If it is determined to continue (including if the user makes a selection related to the new location to be presented in box 617), the routine returns to box 605 to await additional instructions or information (or proceeds directly to box 612 if the user makes a selection related to the new location to be presented in box 617), and if not, proceeds to step 699 and ends.

[0086] The non-exclusive exemplary implementations described herein are further described in the following clauses.

[0087] A01. A computer-implemented method for performing automated operations on one or more computing devices, comprising:

[0088] The one or more computing devices obtain a first panoramic image and a second panoramic image captured in a corresponding first and second region of a room in a building, wherein each of the first panoramic image and the second panoramic image is in an equal rectangular format and has visual coverage of at least some of the walls, floors and ceilings of the room;

[0089] One or more first non-panoramic images are generated from the first panoramic image by the one or more computing devices and by using first acquisition pose information for capturing the first panoramic image, the one or more first non-panoramic images being in a linear format and each including a subset of the visual coverage of the first panoramic image.

[0090] The estimated second acquisition pose information for capturing the second panoramic image is adjusted by the one or more computing devices during multiple iterations until the visual overlay alignment of the first panoramic image and the second panoramic image is achieved, wherein each of the multiple iterations includes:

[0091] One or more second non-panoramic images for the current iteration of the multiple iterations are generated from the second panoramic image by the one or more computing devices and by using the current value of the estimated second acquisition pose information, the one or more second non-panoramic images being in a linear format and each including a subset of the visual coverage of the second panoramic image;

[0092] The difference between a first location and a second location of a structural feature identified in the room is determined by the one or more computing devices using one or more trained neural networks, wherein the first location is identified from the analysis of the one or more first non-panoramic images and the first panoramic image, and wherein the second location is identified from the analysis of the one or more second non-panoramic images and the second panoramic image used in the current iteration; and

[0093] If the determined difference meets one or more limiting criteria, the one or more computing devices determine that the visual coverage of the first panoramic image and the second panoramic image is aligned; otherwise, based on the determined difference, the current value of the estimated second acquisition pose information of the second panoramic image is updated and the next iteration in the multiple iterations is initiated.

[0094] The estimated room shape is generated by the one or more computing devices based on a combination of the aligned visual overlays of the first panoramic image and the second panoramic image, including using a first position from the one or more first non-panoramic images and the first panoramic image, and also using the one or more second non-panoramic images from the last iteration of the multiple iterations and a second position from the second panoramic image, wherein the estimated room shape is a fully enclosed shape having planar surfaces connecting at least the walls of the room; and

[0095] The estimated room shape is displayed by the one or more computing devices.

[0096] A02. A computer-implemented method for performing automated operations on one or more computing devices, comprising:

[0097] The one or more computing devices obtain a first panoramic image captured by an image acquisition device in a first area of ​​a room in a building and a second panoramic image captured by the image acquisition device in a second area of ​​the room, wherein each of the first panoramic image and the second panoramic image is in a spherical format and includes 360-degree horizontal visual coverage around a vertical axis, and wherein each of the first panoramic image and the second panoramic image individually has visual coverage of a subset of the walls, floor and ceiling of the room, and collectively has visual coverage of all the walls, floor and ceiling;

[0098] The one or more computing devices generate a plurality of first straight line images from the first panoramic image by using first acquisition pose information of the image acquisition device during the acquisition of the first panoramic image. The first straight line images are in perspective format and each includes a visual overlay of a subset of the first panoramic image. The plurality of first straight line images include a first ceiling image that includes a visual overlay of the ceiling of the room and a first floor image that includes a visual overlay of the floor of the room.

[0099] The one or more computing devices adjust the estimated second acquisition pose information used by the image acquisition device during the acquisition of the second panoramic image during multiple iterations until the visual overlay of the first and second panoramic images is aligned, including for each of the multiple iterations:

[0100] The one or more computing devices generate a plurality of second straight line images for the current iteration of the multiple iterations by using the current value of the estimated second acquisition pose information and from the second panoramic image. The second straight line images are in perspective format and each includes a visual overlay of a subset of the second panoramic image. The plurality of second straight line images include a second ceiling image that includes a visual overlay of the ceiling of the room and a second floor image that includes a visual overlay of the floor of the room.

[0101] The difference between a first location and a second location of a structural feature identified in the room is determined by the one or more computing devices using at least one trained convolutional neural network, wherein the first location is identified from the analysis of a first straight-line image and a first panoramic image, and wherein the second location is identified from the analysis of a second straight-line image and a second panoramic image used for the current iteration; and

[0102] If the determined difference is below a defined threshold, the one or more computing devices determine that the visual coverage of the first panoramic image and the second panoramic image is aligned; otherwise, based on the determined difference, the current value of the estimated second acquisition pose information of the second panoramic image is updated and the next iteration in the multiple iterations is initiated.

[0103] An estimated three-dimensional (“3D”) room shape is generated by the one or more computing devices based on a combination of aligned visual overlays of the first panoramic image and the second panoramic image, including using a first position from the one or more first non-panoramic images and the first panoramic image, and also using the one or more second non-panoramic images from the last iteration of the multiple iterations and a second position from the second panoramic image, wherein the estimated 3D room shape is a fully enclosed shape having planar surfaces connecting the walls and the floor and ceiling of the room; and

[0104] The one or more computing devices display at least a portion of the floor plan of the building, including the generated estimated 3D room shapes of the rooms.

[0105] A03. A computer-implemented method for performing automated operations on one or more computing devices, comprising:

[0106] The one or more computing devices obtain a first panoramic image and a second panoramic image captured in a corresponding first and second region of the room, wherein each of the first panoramic image and the second panoramic image is in a spherical format and has visual coverage of at least some of the walls of the room;

[0107] One or more first non-panoramic images are generated from the first panoramic image by the one or more computing devices and by using first acquisition pose information for capturing the first panoramic image, each of the one or more first non-panoramic images being in perspective format and including a subset of the visual coverage of the first panoramic image.

[0108] The estimated second acquisition pose information for capturing the second panoramic image is adjusted by the one or more computing devices during multiple iterations until the visual overlay of the first and second panoramic images is aligned, wherein each of the multiple iterations includes:

[0109] One or more second non-panoramic images are generated from the second panoramic image by the one or more computing devices and by using the current value of the estimated second acquisition pose information for the current iteration in the multiple iterations, each of the one or more second non-panoramic images being in perspective format and including visual coverage of a subset of the second panoramic image;

[0110] The difference between a first location and a second location of a feature identified in the room is determined by the one or more computing devices, wherein the first location is identified from visual data of the one or more first non-panoramic images and the first panoramic image, and wherein the second location is identified from visual data of the one or more second non-panoramic images and the second panoramic image used in the current iteration; and

[0111] If the determined difference for the current iteration meets one or more limiting criteria, then the one or more computing devices determine that the visual coverage of the first panoramic image and the second panoramic image is aligned; otherwise, based on the determined difference, the current value of the estimated second acquisition pose information of the second panoramic image is updated and the next iteration in the multiple iterations is initiated.

[0112] The estimated room shape is generated by the one or more computing devices based on a combination of the aligned visual overlays of the first panoramic image and the second panoramic image, including using a first position from the one or more first non-panoramic images and the first panoramic image, and also using the one or more second non-panoramic images from the last iteration of the multiple iterations and a second position from the second panoramic image; and

[0113] Information about the room is provided by the one or more computing devices, including a generated estimated room shape, so that the provided information can be displayed.

[0114] A04. A computer-implemented method for performing automated operations on one or more computing devices, comprising:

[0115] Obtain a first panoramic image and a second panoramic image captured in a corresponding first and second region of the room, wherein each of the first panoramic image and the second panoramic image is in spherical format and has visual coverage of at least some of the walls of the room;

[0116] One or more first non-panoramic images are generated from the first panoramic image using first acquisition pose information for capturing the first panoramic image. Each of the one or more first non-panoramic images is in perspective format and includes visual coverage of a subset of the first panoramic image.

[0117] During one or more iterations until the visual overlay of the first and second panoramic images is aligned, the estimated second acquisition pose information for capturing the second panoramic image is adjusted, and for each of the one or more iterations, the following is included:

[0118] One or more second non-panoramic images are generated from the second panoramic image using the current value of the estimated second acquisition pose information for the current iteration in one or more iterations, each of the one or more second non-panoramic images being in perspective format and including visual coverage of a subset of the second panoramic image;

[0119] One or more trained neural networks are used to determine the difference between a first location and a second location of a feature identified in the room, wherein the first location is identified from visual data of the one or more first non-panoramic images and the first panoramic image, and wherein the second location is identified from visual data of the one or more second non-panoramic images and the second panoramic image used in the current iteration; and

[0120] If the determined difference meets one or more limiting criteria, then the visual coverage of the first panoramic image and the second panoramic image is determined to be aligned; otherwise, the current value of the estimated second acquisition pose information of the second panoramic image is updated based on the determined difference and the next iteration is initiated.

[0121] Generating one or more estimated shapes of one or more structural elements that are part of at least one of the walls of the room based on a combination of the visual overlays of the alignment of the first panoramic image and the second panoramic image, including using a first position from the one or more first non-panoramic images and the first panoramic image, and also using the one or more second non-panoramic images from the last iteration of the one or more iterations and a second position from the second panoramic image; and

[0122] Provide information about the room, including the generated estimated room shape.

[0123] A05. A computer-implemented method as described in any one of clauses A01 to A04, wherein the image acquisition device has one or more inertial measurement unit (IMU) sensors for capturing motion data, and wherein the method further comprises:

[0124] The first acquisition pose information is determined by the one or more computing devices based at least in part on first motion data captured by the one or more IMU sensors during the acquisition of the first panoramic image and on visual data from the first panoramic image; and

[0125] The estimated initial value of the second acquisition pose information is determined by the one or more computing devices based at least in part on second motion data captured by the one or more IMU sensors during the acquisition of the second panoramic image and on visual data from the second panoramic image.

[0126] And wherein, for the first iteration of the multiple iterations, the current value of the estimated second acquisition posture information is the determined initial value of the estimated second acquisition posture information.

[0127] A06. A computer-implemented method as described in any one of clauses A01 to A05, wherein, for each of the plurality of iterations, determining the difference between the first and second positions of the structural feature identified in the room for the iteration comprises: determining a change in position and rotation of the estimated second acquisition posture information for the iteration relative to the first acquisition posture information, and updating the current value of the estimated second acquisition posture information for the iteration comprises: updating the current value based on the determined change in position and rotation for the iteration, such that the centers of the second ceiling image and the second floor image generated in the next iteration are shifted to correspond to the determined change in position and rotation for the iteration.

[0128] A07. A computer-implemented method as described in any one of clauses A01 to A06, wherein, for each of the plurality of iterations, determining the difference between the first and second positions of the structural features identified in the room for the iteration comprises: determining the alignment difference between the floor and the ceiling of the room for the iteration, and updating the current value of the estimated second acquisition posture information for the iteration comprises: determining a new current value of the estimated second acquisition posture information such that the second ceiling image and the second floor image generated in the next iteration are shifted to reduce the alignment difference between the floor and the ceiling.

[0129] A08. A computer-implemented method as described in any one of clauses A01 to A07, wherein the first panoramic image and the second panoramic image include visual coverage of at least some of the floor and ceiling of the room, wherein the one or more second non-panoramic images generated for each of the multiple iterations include visual coverage of at least some of the floor and ceiling, and wherein the method further comprises, for each of the multiple iterations:

[0130] Features in the room are identified in part based on one or more second non-panoramic images generated for the iteration and corresponding to one or more of the walls, floors, or ceilings of the room.

[0131] And wherein determining the difference between the first and second positions of the features identified in the room for the iteration includes: determining the change in position and rotation of the estimated second acquisition pose information of the iteration relative to the first acquisition pose information, and wherein updating the current value of the estimated second acquisition pose information for the iteration includes: updating the current value based on the determined change in position and rotation for the iteration, such that the center of the one or more second non-panoramic images generated in the next iteration is shifted to correspond to the determined change in position and rotation for the iteration.

[0132] A09. A computer-implemented method as described in any one of clauses A01 to A08, wherein the first panoramic image and the second panoramic image include visual coverage of at least some of the floor and ceiling of the room, wherein the one or more second non-panoramic images generated for each of the multiple iterations include visual coverage of at least some of the floor and ceiling, and wherein the method further comprises, for each of the multiple iterations:

[0133] Features in the room are identified in part based on the one or more second non-panoramic images generated for the iteration.

[0134] And wherein determining the difference between the first and second positions of the features identified in the room for the iteration includes: determining the alignment difference of the floor and the ceiling of the room for the iteration, and wherein updating the current value of the estimated second acquisition pose information for the iteration includes: determining a new current value of the estimated second acquisition pose information such that the one or more second non-panoramic images generated in the next iteration include visual overlay of at least some of the floor and the ceiling, the visual overlay being shifted to reduce the alignment difference of the floor and the ceiling.

[0135] A10. A computer-implemented method as described in any one of clauses A01 to A09, wherein the first panoramic image and the second panoramic image each comprise a 360-degree horizontal visual overlay around a vertical axis, wherein adjusting the estimated second acquisition pose information and generating the estimated room shape are performed without using any depth information from any depth sensing sensor regarding distances from the first and second regions of the room to surrounding surfaces, wherein generating the one or more first non-panoramic images comprises: using the first acquisition pose information to generate a first projected ceiling image comprising a visual overlay of the room's ceiling, and generating a first projected floor image comprising a visual overlay of the room's floor, wherein generating the one or more second non-panoramic images for each of the multiple iterations comprises: using the current value for the estimated second acquisition pose information of the iteration to generate a second projected ceiling image comprising a visual overlay of the room's ceiling, and generating a second projected floor image comprising a visual overlay of the room's floor, and wherein the method further comprises:

[0136] If the determined difference is below a defined threshold using the one or more computing devices and for each of the multiple iterations, then the determined difference for the iteration is determined to satisfy the one or more defined criteria.

[0137] The room shape is estimated in first three-dimensional (3D) by the one or more computing devices from the visual data of the first panoramic image;

[0138] The room shape is estimated in 3D by the one or more computing devices from the visual data of the second panoramic image; and

[0139] The one or more computing devices combine the first 3D estimated room shape and the second 3D estimated room shape using the aligned visual overlay of the first panoramic image and the second panoramic image to generate a generated estimated room shape of the room, wherein the generated estimated room shape of the room is a 3D shape.

[0140] A11. A computer-implemented method as described in any one of clauses A01 to A10, wherein the features identified in the room for the multiple iterations include structural elements that are part of at least one of the walls, floor, or ceiling of the room, wherein determining the difference between the first and second positions of the features identified in the room for each of at least one of the multiple iterations comprises: using one or more trained neural networks to analyze information about structural elements identified from visual data of the one or more second non-panoramic images and the one or more first non-panoramic images and the first panoramic image and the second panoramic image for the iteration, and wherein providing the information about the room comprises: the one or more computing devices transmitting a generated estimated room shape of the room via one or more networks and transmitting it to one or more client devices such that the generated estimated room shape of the room is displayed on the one or more client devices.

[0141] A12. A computer-implemented method as described in any one of clauses A01 to A11, wherein the visual overlay of the first panoramic image and the second panoramic image includes at least some of the ceiling and floor of the room, wherein the spherical format of the first panoramic image and the second panoramic image is an equal rectangular format, wherein the perspective format of the one or more first non-panoramic images and the one or more second non-panoramic images for each of the multiple iterations is a linear format, wherein the room is one of a plurality of rooms in a building, wherein generating an estimated shape of the room includes: generating a fully enclosed three-dimensional (3D) room shape, the room shape including and further including constructing at least a partial floor plan of the building including the generated 3D room shape, and wherein improving the information about the room includes: providing at least a partial floor plan of the constructed building.

[0142] A13. A computer-implemented method as described in any one of clauses A01 to A12, wherein the first panoramic image and the second panoramic image include visual coverage of at least some of the floor and ceiling of the room, wherein the one or more second non-panoramic images generated for each of the multiple iterations include visual coverage of at least some of the floor and ceiling, wherein the automated operation further includes identifying features in the room for each of the multiple iterations in part based on the one or more second non-panoramic images generated for the iterations and corresponding to one or more of the walls or the floor or the ceiling of the room, and wherein for each of the multiple iterations:

[0143] Determining the difference between the first and second positions of the features identified in the room for the iteration includes: determining the change in position and rotation of the current value of the estimated second acquisition posture information of the iteration relative to the first acquisition posture information; and

[0144] Updating the current value of the estimated second acquisition pose information for the iteration includes updating the current value based on determined changes in position and rotation for the iteration, such that the center of the one or more second non-panoramic images generated in the next iteration is shifted to correspond to the determined changes in position and rotation for the iteration.

[0145] A14. A computer-implemented method as described in any one of clauses A01 to A13, wherein the first panoramic image and the second panoramic image include visual coverage of at least some of the floor and ceiling of the room, wherein the one or more second non-panoramic images generated for each of the multiple iterations include visual coverage of at least some of the floor and ceiling, wherein the automated operation further includes identifying features in the room for each of the multiple iterations, in part based on the one or more second non-panoramic images generated for the iterations, and wherein for each of the multiple iterations:

[0146] Determining the difference between the first and second locations of the features identified in the room for the iteration includes: the difference in alignment between the floor and the ceiling of the room for the iteration; and

[0147] Updating the current value of the estimated second acquisition pose information for the iteration includes: determining a new current value of the estimated second acquisition pose information such that the one or more second non-panoramic images generated in the next iteration include visual overlay of at least some of the floor and the ceiling, the visual overlay being shifted to reduce the alignment difference between the floor and the ceiling.

[0148] A15. A computer-implemented method as described in any one of clauses A01 to A14, wherein acquiring the first panoramic image and the second panoramic image is performed by an image acquisition device having one or more inertial measurement unit (IMU) sensors, and includes using the one or more IMU sensors to capture motion data, and wherein the method further comprises:

[0149] The first acquisition pose information is determined by the one or more computing devices based at least in part on first motion data captured by the one or more IMU sensors during the acquisition of the first panoramic image and on visual data from the first panoramic image; and

[0150] The estimated initial value of the second acquisition pose information is determined by the one or more computing devices based at least in part on second motion data captured by the one or more IMU sensors during the acquisition of the second panoramic image and on visual data from the second panoramic image.

[0151] And wherein, for the first iteration of the multiple iterations, the current value of the estimated second acquisition posture information is the determined initial value of the estimated second acquisition posture information.

[0152] A16. A computer-implemented method as described in any one of clauses A01 to A15, wherein acquiring the first panoramic image and the second panoramic image and generating the one or more first non-panoramic images, adjusting the estimated second acquisition pose information, and generating the estimated room shape are performed for each of a plurality of rooms in a building, wherein generating the estimated room shape for each of the plurality of rooms comprises: analyzing visual data from the first panoramic image and the second panoramic image captured in the room to identify one or more locations of one or more wall openings from the room to one or more other rooms, and wherein the automated operation further comprises assembling the generated estimated room shapes of the plurality of rooms at least in part based on the identified locations of the wall openings of the plurality of rooms to generate at least a partial floor plan by the one or more computing devices.

[0153] A17. The computer-implemented method as described in any one of clauses A01 to A16,

[0154] The automated operation further includes receiving first information from one or more users to specify at least one initial value of the first acquisition posture information or the estimated second acquisition posture information, the initial value being used as the current value of the estimated second acquisition posture for the first iteration in the multiple iterations; and / or

[0155] Determining the difference between the first and second positions of the feature identified in the room for each of at least one of the multiple iterations includes: receiving second information from one or more users to specify at least some of the first and second positions of the feature identified in the room for the iteration; and / or

[0156] Determining the difference between the first and second positions of the feature identified in the room for each of at least one of the multiple iterations includes: receiving third information from one or more users to specify at least some of the differences between the first and second positions of the feature identified in the room for the iteration; and / or

[0157] Updating the current value of the estimated second acquisition pose information for at least one of the multiple iterations includes: receiving fourth information from one or more users to specify at least some of the current values ​​of the estimated second acquisition pose information for the iteration; and / or

[0158] The process of reviewing the estimated room shape of the room includes: obtaining a first estimated room shape of the room generated from visual data of the first panoramic image, and further includes obtaining a second estimated room shape of the room generated from visual data of the second panoramic image, and further includes receiving fifth information from one or more users to specify how to combine at least some of the first estimated room shape and the second estimated room shape to generate the generated estimated room shape of the room.

[0159] A18. A computer-implemented method as described in clause A17, wherein the automated operation further includes the use of at least one of the received first information, the received second information, the received third information, the received fourth information, or the received fifth information by the one or more computing devices during each adjustment of the estimated second acquisition posture information for at least one of the multiple iterations, and providing constraints.

[0160] A19. A computer-implemented method as described in any one of clauses A01 to A18, wherein generating the one or more first non-panoramic images comprises: using the first acquisition posture information to generate first projected ceiling visual data including visual coverage of the ceiling of the room and first projected floor visual data including visual coverage of the floor of the room, and further comprising adjusting the size of at least one of the ceiling or the floor in the generated first projected ceiling visual data and first projected floor visual data to match each other using a first height of the image acquisition device above the floor of the room during the capture of the first panoramic image, and wherein generating the one or more second non-panoramic images for each of the plurality of iterations comprises: using the current value of the estimated second acquisition posture information for the iteration to generate second projected ceiling visual data including visual coverage of the ceiling of the room and second projected floor visual data including visual coverage of the floor of the room, and further comprising adjusting the size of at least one of the ceiling or the floor in the generated second projected ceiling visual data and second projected floor visual data to match each other using a second height of the image acquisition device above the floor of the room during the capture of the second panoramic image.

[0161] A20. A computer-implemented method as described in any one of clauses A01 to A19, wherein the automated operation further comprises:

[0162] If the determined difference is below a defined threshold using the one or more computing devices and for each of the multiple iterations, then the determined difference for the iteration is determined to satisfy the one or more defined criteria.

[0163] The one or more computing devices generate a first estimated room shape from the visual data of the first panoramic image;

[0164] The one or more computing devices generate a second estimated room shape of the room from the visual data of the second panoramic image; and

[0165] The first estimated room shape and the second estimated room shape are combined by the one or more computing devices using the aligned visual overlay of the first panoramic image and the second panoramic image to generate the generated estimated room shape of the room.

[0166] A21. A computer-implemented method as described in any one of clauses A01 to A20, wherein the first panoramic image and the second panoramic image each comprise 360-degree horizontal visual coverage around a vertical axis, and wherein adjusting the estimated second acquisition pose information and generating the estimated room shape are performed without using any depth information from any depth sensing sensor regarding distances from the first and second regions of the room to surrounding surfaces.

[0167] A22. A computer-implemented method as described in any one of clauses A01 to A21, wherein the one or more iterations comprise multiple iterations, wherein the features identified in the room for the multiple iterations comprise a plurality of structural elements that are part of at least one of the walls, floor, or ceiling of the room, wherein determining the difference between the first and second positions of the features identified in the room for each of at least one of the multiple iterations comprises: analyzing information about at least some of the plurality of structural elements, the plurality of structural elements being identified from visual data of the one or more second non-panoramic images and the one or more first non-panoramic images and the first panoramic image and the second panoramic image for the iteration, and wherein providing the information about the room comprises: sending one or more estimated shapes generated via one or more networks to one or more client devices such that the one or more estimated shapes generated are displayed on the one or more client devices.

[0168] A23. The computer-implemented method as described in clause A22, further comprising generating an estimated room shape of the room based on a combination of the visual overlays of the alignment of the first panoramic image and the second panoramic image, including using a first position from the one or more first non-panoramic images and the first panoramic image, and also using the one or more second non-panoramic images from the last iteration of the one or more iterations and the second position from the second panoramic image, wherein transmitting the generated one or more estimated shapes further comprises: transmitting the generated estimated room shape of the room such that the generated one or more estimated shapes are displayed on a display of the generated estimated room shape of the room on the one or more client devices.

[0169] A24. A computer-implemented method as described in clause A23, wherein the visual overlay of the first panoramic image and the second panoramic image includes at least some of the ceiling and floor of the room, wherein the spherical format of the first panoramic image and the second panoramic image is an isorectangular format, wherein the perspective format of the one or more first non-panoramic images and the one or more second non-panoramic images for each of the one or more iterations is a linear format, wherein the room is one of a plurality of rooms in a building, wherein generating an estimated shape of the room includes: generating a fully enclosed three-dimensional (3D) room shape, the room shape including the connecting plane surfaces of the walls and the floor and the ceiling of the room, and further includes constructing at least a partial floor plan of the building including the generated 3D room shape, and wherein providing the information about the room includes: providing the constructed at least a partial floor plan of the building.

[0170] A25. A computer-implemented method as described in any one of clauses A01 to A24, wherein the one or more iterations comprise multiple iterations, wherein the first panoramic image and the second panoramic image comprise visual coverage of at least some of the floor and ceiling of the room, wherein the one or more second non-panoramic images generated for each of the multiple iterations comprise visual coverage of at least some of the floor and ceiling, wherein the automated operation further comprises identifying features in the room for each of the multiple iterations in part based on the one or more second non-panoramic images generated for the iterations and corresponding to one or more of the walls or the floor or the ceiling of the room, and wherein for each of the multiple iterations:

[0171] Determining the difference between the first and second positions of the features identified in the room for the iteration includes: determining the change in position and rotation of the current value of the estimated second acquisition posture information of the iteration relative to the first acquisition posture information; and

[0172] Updating the current value of the estimated second acquisition pose information for the iteration includes updating the current value based on determined changes in position and rotation for the iteration, such that the center of the one or more second non-panoramic images generated in the next iteration is shifted to correspond to the determined changes in position and rotation for the iteration.

[0173] A26. A computer-implemented method as described in any one of clauses A01 to A25, wherein the one or more iterations comprise multiple iterations, wherein the first panoramic image and the second panoramic image comprise visual coverage of at least some of the floor and ceiling of the room, wherein the one or more second non-panoramic images generated for each of the multiple iterations comprise visual coverage of at least some of the floor and ceiling, wherein the automated operation further comprises identifying features in the room for each of the multiple iterations, in part based on the one or more second non-panoramic images generated for the iterations, and wherein for each of the multiple iterations:

[0174] Determining the difference between the first and second locations of the features identified in the room for the iteration includes: the difference in alignment between the floor and the ceiling of the room for the iteration; and

[0175] Updating the current value of the estimated second acquisition pose information for the iteration includes: determining a new current value of the estimated second acquisition pose information such that the one or more second non-panoramic images generated in the next iteration include visual overlay of at least some of the floor and the ceiling, the visual overlay being shifted to reduce the alignment differences between the floor and the ceiling.

[0176] A27. A computer-implemented method as described in any one of clauses A01 to A26, wherein generating one or more estimated shapes of the structural elements in the walls of the room includes at least one of: generating an estimated door shape for a door in at least one of the walls of the room, or generating an estimated window shape for a window in at least one of the walls of the room, or generating an estimated opening shape for a non-door and non-window opening in at least one of the walls of the room.

[0177] A28. A computer-implemented method as described in any one of clauses A01 to A27, wherein generating the one or more estimated shapes of the one or more structural elements in the walls of the room comprises: estimating the position of each of the one or more estimated shapes of the one or more structural elements within the estimated shape of the room.

[0178] A29. A computer-implemented method comprising multiple steps to perform automated operations, said automated operations being implemented substantially as described herein.

[0179] B01. A non-transitory computer-readable medium storing executable software instructions and / or other stored content, said executable software instructions and / or other stored content causing one or more computing systems to perform automated operations, said automated operations implementing the method of any one of clauses A01 to A29.

[0180] B02. A non-transitory computer-readable medium storing executable software instructions and / or other stored content, said executable software instructions and / or other stored content causing one or more computing systems to perform automated operations, said automated operations being implemented substantially as described herein.

[0181] C01. One or more computing systems, including one or more hardware processors and one or more memories storing instructions, which, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations, said automated operations implementing the method of any one of clauses A01 to A29.

[0182] C02. One or more computing systems, including one or more hardware processors and one or more memories storing instructions that, when executed by at least one of the one or more hardware processors, cause the one or more computing systems to perform automated operations, said automated operations being implemented substantially as described herein.

[0183] D01. A computer program, when run on a computer, is adapted to perform the method of any one of clauses A01 to A29.

[0184] This document describes aspects of the disclosure with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions. It will be further understood that in some embodiments, the functionality provided by the routines discussed above can be provided in alternative ways, such as being divided into more routines or merged into fewer routines. Similarly, in some embodiments, the illustrated routines can provide more or less functionality than described, such as when other illustrated routines are correspondingly modified to omit or include such functionality, or when the amount of functionality provided changes. Additionally, while various operations may be described as being performed in a particular manner (e.g., serial or parallel or synchronous or asynchronous) and / or in a particular order, in other embodiments, operations may be performed in other orders and other manners. Any data structures discussed above can also be structured in different ways, such as by dividing a single data structure into multiple data structures and / or by merging multiple data structures into a single data structure. Similarly, in some implementations, the illustrated data structure may store more or less information than the described information, such as when other illustrated data structures are correspondingly modified to omit or include such information, or when the amount or type of the stored information changes.

[0185] Based on the foregoing, it will be understood that although specific embodiments have been described herein for illustrative purposes, various modifications can be made without departing from the spirit and scope of the invention. Therefore, the invention is not limited except by the corresponding claims and the elements recited in those claims. Furthermore, although certain aspects of the invention may be presented in certain claims at certain times, the inventors contemplate various aspects of the invention in any available claim form. For example, although only some aspects of the invention may be described as embodied in a computer-readable storage medium at certain times, other aspects may also be embodied in the same way.

Claims

1. A non-transitory computer-readable medium having stored content that causes a computing system to perform automated operations, the automated operations including at least: The computing system obtains a first panoramic image and a second panoramic image captured in a corresponding first area and a second area of ​​a room in a building, wherein each of the first panoramic image and the second panoramic image is in an equal rectangular format and has visual coverage of at least some of the walls of the room; The computing system generates one or more first non-panoramic images from the first panoramic image by using first acquisition pose information for capturing the first panoramic image. Each of the one or more first non-panoramic images is in a linear format and includes visual coverage of a subset of the first panoramic image. The computational system adjusts the estimated second acquisition pose information for capturing the second panoramic image during multiple iterations until the visual overlay alignment of the first and second panoramic images is achieved, with each of the multiple iterations including: The computing system generates one or more second non-panoramic images from the second panoramic image for the current iteration in the multiple iterations by using the current value of the estimated second acquisition pose information. Each of the one or more second non-panoramic images is in a linear format and includes visual coverage of a subset of the second panoramic image. The computing system determines the difference between a first location and a second location of a feature identified in the room, wherein the first location is identified from visual data of one or more first non-panoramic images and the first panoramic image, and wherein the second location is identified from visual data of the one or more second non-panoramic images and the second panoramic image used in the current iteration; and If the difference determined for the current iteration meets one or more limiting criteria, the computing system determines that the visual coverage of the first panoramic image and the second panoramic image is aligned; otherwise, based on the determined difference, the current value of the estimated second acquisition pose information of the second panoramic image is updated, and the next iteration in the multiple iterations is initiated. The computing system generates an estimated room shape based on a combination of aligned visual overlays of the first panoramic image and the second panoramic image, including using a first position from the one or more first non-panoramic images and the first panoramic image, and also using the one or more second non-panoramic images from the last iteration of the multiple iterations and a second position from the second panoramic image, wherein the estimated room shape includes planar surfaces for the connection of at least the walls of the room; and The computing system provides information about the room, including a generated estimated room shape, to enable the display of the provided information.

2. The non-transitory computer-readable medium as claimed in claim 1, wherein, The features identified in the room for the multiple iterations include structural elements that are part of at least one of the walls, floor, or ceiling of the room, wherein determining the difference between the first and second positions of the features identified in the room for each of at least one of the multiple iterations includes: using one or more trained neural networks to analyze information about structural elements identified from visual data of the one or more second non-panoramic images and the one or more first non-panoramic images and the first panoramic image and the second panoramic image for the iteration, and wherein providing information about the room includes: the computing system sending a generated estimated room shape of the room to one or more client devices via one or more networks to display the generated estimated room shape of the room on the one or more client devices.

3. The non-transitory computer-readable medium as claimed in claim 1, wherein, The visual overlay of the first panoramic image and the second panoramic image includes at least some of the ceiling and floor of the room, wherein the room is one of a plurality of rooms in a building, wherein generating an estimated shape of the room includes generating a fully enclosed three-dimensional (3D) room shape, and further includes constructing at least a partial floor plan of the building including the generated 3D room shape, and wherein providing information about the room includes providing the constructed at least a partial floor plan of the building.

4. The non-transitory computer-readable medium as claimed in claim 1, wherein, The first panoramic image and the second panoramic image include visual coverage of at least some of the floor and ceiling of the room, wherein the one or more second non-panoramic images generated for each of the multiple iterations include visual coverage of at least some of the floor and ceiling, wherein the automated operation further includes, in part, identifying features in the room for each of the multiple iterations based on the one or more second non-panoramic images generated for the iterations and corresponding to one or more of the walls, floor, or ceiling of the room, and wherein, for each of the multiple iterations: Determining the difference between the first and second positions of the features identified in the room for the iteration includes: determining the change in position and rotation of the current value of the estimated second acquisition posture information for the iteration relative to the first acquisition posture information; and Updating the current value of the estimated second acquisition pose information used in the iteration includes updating the current value based on determined changes in position and rotation used in the iteration, such that the center of the one or more second non-panoramic images generated in the next iteration is shifted to correspond to the determined changes in position and rotation used in the iteration.

5. The non-transitory computer-readable medium as claimed in claim 1, wherein, The first panoramic image and the second panoramic image include visual coverage of at least some of the floor and ceiling of the room, wherein the one or more second non-panoramic images generated for each of the multiple iterations include visual coverage of at least some of the floor and ceiling, wherein the automated operation further includes: identifying features in the room for each of the multiple iterations, in part based on the one or more second non-panoramic images generated for the iterations, and wherein for each of the multiple iterations: Determining the difference between the first and second locations of the features identified in the room for the iteration includes: determining the alignment difference between the floor and the ceiling of the room for the iteration; and Updating the current value of the estimated second acquisition pose information used in the iteration includes: determining a new current value for the estimated second acquisition pose information such that the one or more second non-panoramic images generated in the next iteration include visual overlay of at least some of the floor and the ceiling, the visual overlay being shifted to reduce the alignment difference between the floor and the ceiling.

6. The non-transitory computer-readable medium as claimed in claim 1, wherein, Acquiring the first panoramic image and the second panoramic image is performed by an image acquisition device having one or more inertial measurement unit (IMU) sensors, and includes using the one or more IMU sensors to capture motion data, wherein the stored content includes software instructions that, when executed by the computing system, cause the computing system to perform further automated operations, the further automated operations including: The computing system determines the first acquisition pose information based at least in part on first motion data captured by the one or more IMU sensors during the acquisition of the first panoramic image, and based on visual data from the first panoramic image; and The computing system determines, at least in part, the initial value of the estimated second acquisition pose information based on second motion data captured by the one or more IMU sensors during the acquisition of the second panoramic image, and based on visual data from the second panoramic image. Wherein, the current value of the estimated second acquisition posture information used in the first iteration of the multiple iterations is the determined initial value of the estimated second acquisition posture information.

7. The non-transitory computer-readable medium as claimed in claim 1, wherein, The process involves acquiring the first panoramic image and the second panoramic image, generating the one or more first non-panoramic images, adjusting the estimated second acquisition pose information, and generating the estimated room shape for each of the plurality of rooms in the building. Generating the estimated room shape for each of the plurality of rooms includes: analyzing visual data from the first and second panoramic images captured in the room to identify one or more locations of one or more wall openings from the room to one or more other rooms; and wherein the automated operation further includes: assembling the generated estimated room shapes of the plurality of rooms by the computing system, at least in part, based on the identified locations of the wall openings of the plurality of rooms, to generate at least a partial floor plan.

8. The non-transitory computer-readable medium as described in claim 1, in, The automated operation further includes receiving first information from one or more users to specify an initial value of the estimated second acquisition posture information or at least one of the first acquisition posture information, the initial value being used as the current value of the estimated second acquisition posture information for the first iteration of the multiple iterations; and / or Determining the difference between the first and second positions of the feature identified in the room for at least one of the multiple iterations includes: receiving second information from one or more users to specify at least some of the first and second positions of the feature identified in the room for the iteration; and / or Determining the difference between the first and second positions of the feature identified in the room for at least one of the multiple iterations includes: receiving third information from one or more users to specify at least some of the differences between the first and second positions of the feature identified in the room for the iteration; and / or Updating the current value of the estimated second acquisition pose information for at least one of the multiple iterations includes: receiving fourth information from one or more users to specify at least some of the current values ​​of the estimated second acquisition pose information for the iteration; and / or The process of generating the estimated room shape of the room includes: obtaining a first estimated room shape of the room generated from visual data of the first panoramic image, and further includes obtaining a second estimated room shape of the room generated from visual data of the second panoramic image, and further includes receiving fifth information from one or more users to specify how to combine at least some of the first estimated room shape and the second estimated room shape to generate the generated estimated room shape of the room.

9. The non-transitory computer-readable medium of claim 8, wherein, The automated operation further includes: the computing system using at least one of the received first information, the received second information, the received third information, the received fourth information, or the received fifth information, and providing constraints during each adjustment of the estimated second acquisition posture information for at least one of the multiple iterations.

10. The non-transitory computer-readable medium of claim 1, wherein, Generating the one or more first non-panoramic images includes: using the first acquisition posture information to generate first projected ceiling visual data including visual coverage of the ceiling of the room, and generating first projected floor visual data including visual coverage of the floor of the room, and further includes, during the capture of the first panoramic image, using an image acquisition device at a first height above the floor of the room, adjusting the size of at least one of the ceiling or the floor in the generated first projected ceiling visual data and the first projected floor visual data to match each other, and wherein generating the one or more second non-panoramic images for each of the multiple iterations includes: using the current value of the estimated second acquisition posture information for the iteration, generating second projected ceiling visual data including visual coverage of the ceiling of the room, and generating second projected floor visual data including visual coverage of the floor of the room, and further includes, during the capture of the second panoramic image, using an image acquisition device at a second height above the floor of the room, adjusting the size of at least one of the ceiling or the floor in the generated second projected ceiling visual data and the second projected floor visual data to match each other.

11. The non-transitory computer-readable medium of claim 1, wherein, The automated operation also includes: By the computing system and for each of the multiple iterations, if the determined difference is below a defined threshold, then the determined difference for the iteration is determined to satisfy one or more defined criteria; The computing system generates a first estimated room shape from the visual data of the first panoramic image; The computing system generates a second estimated room shape from the visual data of the second panoramic image; and The computing system combines the first estimated room shape and the second estimated room shape using the aligned visual overlay of the first panoramic image and the second panoramic image to generate the generated estimated room shape of the room.

12. The non-transitory computer-readable medium of claim 1, wherein, The first panoramic image and the second panoramic image each include 360-degree horizontal visual coverage around a vertical axis, and wherein adjusting the estimated second acquisition pose information and generating the estimated room shape are performed without using any depth information from any depth sensing sensor regarding the distance from the first and second regions of the room to the surrounding surfaces.

13. A system comprising: One or more hardware processors of one or more computing devices; as well as One or more memories storing instructions that, when executed by at least one of the one or more hardware processors, cause at least one of the one or more computing devices to perform an automated operation, the automated operation comprising at least: Obtain a first panoramic image and a second panoramic image captured in a corresponding first and second region of a room in a building, wherein each of the first panoramic image and the second panoramic image is in an equal rectangular format and has visual coverage of at least some of the walls of the room. By using first acquisition pose information for capturing the first panoramic image, one or more first non-panoramic images are generated from the first panoramic image, each of the one or more first non-panoramic images being in a linear format and including visual coverage of a subset of the first panoramic image. During multiple iterations until the visual overlay of the first panoramic image and the second panoramic image is aligned, the estimated second acquisition pose information for capturing the second panoramic image is adjusted, including for each of the multiple iterations: By using the current value of the estimated second acquisition pose information, one or more second non-panoramic images are generated from the second panoramic image for the current iteration in the multiple iterations, each of the one or more second non-panoramic images being in a linear format and including visual coverage of a subset of the second panoramic image; One or more trained neural networks are used to determine the difference between a first location and a second location of a feature identified in the room, wherein the first location is identified from visual data of the one or more first non-panoramic images and the first panoramic image, and wherein the second location is identified from visual data of the one or more second non-panoramic images and the second panoramic image used in the current iteration; and If the determined difference meets one or more qualifying criteria, then the visual coverage of the first panoramic image and the second panoramic image is determined to be aligned; otherwise, the current value of the estimated second acquisition pose information of the second panoramic image is updated based on the determined difference, and the next iteration is initiated. Based on the combination of the aligned visual overlays of the first panoramic image and the second panoramic image, generate one or more estimated shapes of one or more structural elements that are part of at least one of the walls of the room, including using the first position from the one or more first non-panoramic images and the first panoramic image, and also using the one or more second non-panoramic images from the last iteration of the multiple iterations and the second position from the second panoramic image; and Provide information about the room, including the generated estimated room shape.

14. The system of claim 13, wherein, Generating one or more estimated shapes of one or more structural elements in the walls of the room includes at least one of the following: generating an estimated door shape for a door in at least one of the walls of the room, or generating an estimated window shape for a window in at least one of the walls of the room, or generating an estimated opening shape for a non-door and non-window opening in at least one of the walls of the room, or estimating the position of each of the one or more estimated shapes of the structural elements within the estimated shape of the room.

15. A computer-implemented method, comprising: A computing device acquires a first panoramic image and a second panoramic image captured in a corresponding first and second region of a room in a building, wherein each of the first panoramic image and the second panoramic image is in an equal rectangular format and has visual coverage of at least some of the walls, floors and ceilings of the room. The computing device generates one or more first non-panoramic images from the first panoramic image by using first acquisition pose information for capturing the first panoramic image. The one or more first non-panoramic images are in a linear format and each includes a subset of the visual coverage of the first panoramic image. The computing device adjusts the estimated second acquisition pose information for capturing the second panoramic image during multiple iterations until the visual overlay of the first and second panoramic images is aligned, including for each of the multiple iterations: The computing device generates one or more second non-panoramic images from the second panoramic image for the current iteration in the multiple iterations by using the current value of the estimated second acquisition pose information. The one or more second non-panoramic images are in a linear format and each includes a subset of the visual coverage of the second panoramic image. The computing device, using one or more trained neural networks, determines a difference between a first location and a second location of a structural feature identified in the room, wherein the first location is identified from the analysis of one or more first non-panoramic images and the first panoramic image, and wherein the second location is identified from the analysis of one or more second non-panoramic images and the second panoramic image used in the current iteration; and If the determined difference meets one or more limiting criteria, the computing device determines that the visual coverage of the first panoramic image and the second panoramic image is aligned; otherwise, based on the determined difference, the current value of the estimated second acquisition pose information of the second panoramic image is updated, and the next iteration of the multiple iterations is initiated. The computing device generates an estimated room shape based on a combination of the aligned visual overlays of the first panoramic image and the second panoramic image, including using a first position from the one or more first non-panoramic images and the first panoramic image, and also using the one or more second non-panoramic images from the last iteration of the multiple iterations and a second position from the second panoramic image, wherein the estimated room shape is a fully enclosed shape having planar surfaces connecting at least the walls of the room; and The estimated room shape is displayed by the computing device.

Images