Room coordinate axis estimation

Abstract

The invention relates to a system and method for estimating room coordinate axes based on computer vision scanning of a room and its content objects and a coordinate axis majority voting process. The invention provides a method comprising: capturing a plurality of images of an indoor environment having one or more walls and a plurality of objects, each wall and object having a surface oriented along a corresponding plane; local coordinate axis orientations of at least a portion of the wall and the object are determined based on information received from the AR engine. The method includes estimating a room coordinate axis of an indoor environment based on a voting process, wherein the room coordinate axis is determined using a majority-matched local coordinate axis orientation; and outputting an indication of the room coordinate axis.

Description

Technical Field

[0001] This disclosure generally relates to systems and methods for estimating room coordinate axes based on computer vision scanning of a room and its contents. Some embodiments may utilize a majority voting process to establish the room coordinate axes. Background Technology

[0002] Digital representations of interior physical structures (such as rooms in residential or commercial buildings) facilitate efficient construction, maintenance, renovation planning, documentation, and underwriting. For example, once a room is digitally captured, additional dimensional details can be extracted from the model without requiring personnel to physically be present or use traditional measuring tools such as tape measures, gauges, or rulers. Therefore, the ability to accurately and efficiently model rooms and their contents helps reduce costs associated with various applications.

[0003] For example, when generating a 3D model of a room, a typical goal is to derive mathematical representations of the room's walls and contents based on the 3D world's orientation, so that the room can be rendered from various perspectives on a 2D computer screen, allowing for measurement and understanding of room details. In most cases, a top-down 2D blueprint-style room view is preferred, aligning the room's coordinate axes with the computer screen's natural coordinate axes, where each pixel can be defined by X / Y coordinates. However, when scanning a room for 3D modeling, objects within the room (such as doors, sofas, beds, tables, etc.) often exhibit offsets relative to each other and from the room's walls, each with its own coordinate axes. This can cause difficulties and confusion when attempting to align the top-down 2D blueprint-style room view with the computer screen's natural coordinate axes.

[0004] A major problem users face when using their mobile devices to scan rooms to generate 3D models is that they need to perceive "normal operation"—that is, the mobile device correctly captures the room and the computer vision (CV) is functioning correctly. If users are unsure whether the scanning process is working properly, they often panic and try to adjust their capture technology to make CV work as expected, which can lead to capture failures and a poor user experience. Summary of the Invention

[0005] In view of this, there is an urgent need for more convenient, robust and accurate systems and methods to extract or estimate accurate room coordinate axes.

[0006] This disclosure includes systems and methods for estimating room coordinate axes based on computer vision (CV) scanning of a room and its contents.

[0007] According to certain exemplary embodiments of the present disclosure, a computer-implemented method is provided, the method comprising: receiving an input command at a mobile computing device to initiate the capture of a visual document of an indoor environment; communicating with an augmented reality (AR) engine via a camera of the mobile computing device to capture multiple images of the indoor environment, wherein the indoor environment includes one or more walls and multiple objects, each wall and object having a surface oriented along a corresponding plane; determining the local coordinate axis orientation of at least a portion of the walls and objects based on information received from the AR engine; estimating the room coordinate axes of the indoor environment based on a voting process, wherein the room coordinate axes are determined using a majority-matching local coordinate axis orientation; and outputting an indication of the room coordinate axes.

[0008] According to certain exemplary embodiments of the present disclosure, a computer system for capturing environmental spatial documents is disclosed. The system includes a mobile computing device, comprising: a camera configured to capture video; one or more processors in communication with the camera; an augmented reality (AR) engine in communication with the one or more processors; a first memory configured to store the captured video; and a second memory storing computer code that causes the one or more processors to perform: receiving an input command at the mobile computing device to initiate the capture of a visual document of an indoor environment; communicating with the AR engine via the camera of the mobile computing device to capture multiple images of the indoor environment, wherein the indoor environment includes one or more walls and multiple objects, each wall and object having a surface oriented along a corresponding plane; determining the local coordinate axis orientation of at least a portion of the walls and objects based on information received from the AR engine; determining the room coordinate axis of the indoor environment based on a voting process, wherein the room coordinate axis is determined using a majority-matching local coordinate axis orientation; and outputting an indication of the room coordinate axis.

[0009] Some exemplary embodiments of this disclosure include a non-transient computer-readable medium storing instructions that, when executed by a processor of a computing device, cause the computing device to perform the following actions: receiving an input command at a mobile computing device to initiate the capture of a visual document of an indoor environment; communicating with an augmented reality (AR) engine via a camera of the mobile computing device to capture multiple images of the indoor environment, wherein the indoor environment includes one or more walls and multiple objects, each wall and object having a surface oriented along a corresponding plane; determining the local coordinate axis orientation of at least a portion of the walls and objects based on information received from the AR engine; estimating the room coordinate axes of the indoor environment based on a voting process, wherein the room coordinate axes are determined using a majority-matching local coordinate axis orientation; and outputting an indication of the room coordinate axes.

[0010] This document describes in detail other embodiments, features, and aspects of the technology disclosed herein, which should be considered as part of the technology claimed in this disclosure. Other embodiments, features, and aspects can be understood by referring to the following detailed description, drawings, and claims. Attached Figure Description

[0011] The following explanation is based on the accompanying drawings, which are not necessarily drawn to scale.

[0012] Figure 1 A top view of a room is shown, illustrating the requirement to determine the room's principal coordinate axis according to certain embodiments of the present disclosure.

[0013] Figure 2 A top view of a room is shown, rotating the coordinate axes of objects within the room relative to the room's wall / floor boundaries, according to certain exemplary embodiments of the present disclosure.

[0014] Figure 3A A top view of a room with overlaid capture paths is shown according to certain exemplary embodiments of the present disclosure.

[0015] Figure 3B Some exemplary embodiments according to the present disclosure are shown. Figure 3A The diagram shows a top view of objects within a room, each assigned a local coordinate axis based on its orientation, and a voting process to estimate the room's coordinate axes based on all assigned coordinate axes.

[0016] Figure 4 An example of a computing device configured to implement an inspection platform according to certain exemplary embodiments of the present disclosure is shown, the inspection platform being designed to generate measurements of a physical space and the objects contained therein.

[0017] Figure 5 An example block diagram is shown of a processing system that can implement at least some of the operations described herein according to certain exemplary embodiments of the present disclosure.

[0018] Figure 6 A flowchart of a method according to certain exemplary embodiments of the present disclosure is shown.

[0019] Those skilled in the art will gain a clearer understanding of the features of the technology described herein by studying the accompanying drawings and specific embodiments. Those skilled in the art will recognize that other alternative embodiments can be used without departing from the principles of this disclosure. Therefore, although specific embodiments are shown in the drawings, various modifications can still be made to the technology disclosed herein. Detailed Implementation

[0020] This disclosure includes systems and methods for improving the estimation of coordinate axes of an interior physical structure or space (e.g., a room in a residential or commercial building) based on a majority voting process. Objects within the room (e.g., tables, chairs, sofas, appliances, etc.) may be individually assigned coordinate axes, which may not match coordinate axes derived from wall / floor boundaries or other objects in the physical structure or space.

[0021] Certain embodiments of the present disclosure can be used to improve the process and / or user experience of capturing computer vision (CV) scans of a room to construct a three-dimensional (3D) model of the room and its content objects, for example, by providing augmented reality (AR) markers of room features and / or content objects so that the user can be assured that the computing / scanning device and the associated CV are functioning correctly.

[0022] Some embodiments of the present disclosure include providing real-time feedback to a user to capture various views of the room and related content objects using spatial information output by an augmented reality frame (also known as an "AR frame") on a computing device used for imaging / scanning the relevant room.

[0023] In some implementations, real-time feedback may include displaying AR markers to represent naturally occurring features within the room. For example, AR lines at the intersection of a physical wall and a physical floor may be overlaid on the screen of the user's mobile device. In some implementations, frames may be drawn and overlaid around detected furniture. In both cases, the displayed lines or frames may be configured to be precisely aligned with the actual angles of the physical objects. Otherwise, if the orientation of the lines or frames does not match the represented physical objects—wall lines appearing to "pass through" the physical walls and / or furniture frames appearing skewed—this reduces the user's confidence in the correct operation of the computing / scanning equipment and the associated computer vision (CV).

[0024] According to certain exemplary embodiments of the present disclosure, estimating room coordinate axes can provide valuable information for representing a room. In some embodiments, one or more AR lines and / or boxes can be "snap" to the estimated room coordinate axes, which can enable AR overlays to provide a better experience of accurately representing the room.

[0025] According to certain exemplary embodiments of the present disclosure, the room scanning and capture process using estimated room coordinate axes can facilitate easier understanding and use in construction, maintenance, renovation planning, documentation, underwriting, and other processes. For example, the present disclosure allows workers to read like a blue screen. Figure 1 This allows for understanding the room or performing virtual measurements. In some implementations, the final display of the room scan can be a top-down 2D blueprint-style view on a computer screen, utilizing the natural coordinate axes provided by the screen itself, where each pixel can be defined by X / Y coordinates. In some implementations, determining the room coordinate axes allows for easy rotation of the room scan so that the room coordinate axes are aligned with the screen's X / Y axes.

[0026] As described in this disclosure, room coordinate axes can be estimated by majority voting on lines detected in the scene. Therefore, most of the physical geometry of the room may be aligned with the estimated room coordinate axes. Thus, according to certain exemplary embodiments of the technology disclosed herein, most virtual measurements performed using scans presented on a computer screen interface may only need to be performed along the X-direction or only along the Y-direction, which simplifies the measurement and makes it easier to export dimensions to downstream programs.

[0027] Various exemplary embodiments of the present disclosure will now be described more fully with reference to the accompanying drawings, wherein only some embodiments of the present disclosure are shown, and not all embodiments. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the specific embodiments set forth herein; rather, these embodiments are provided to make the disclosure comprehensive and complete, and to convey the scope of protection of the present disclosure to those skilled in the art.

[0028] Figure 1 A top view of room 100 is shown, and its principal coordinate axis 102 needs to be determined. For this room 100, for example, the direction of line 104 in a single image of sofa 106 in room 100 can be used as an estimate of the principal coordinate axis. Similarly, according to certain embodiments of the present disclosure, for example, wall / floor boundary lines 108 in a single image of corner 110 of room 100 can be used as an estimate of the principal coordinate axis and / or affect the confidence level of coordinate axis estimates derived from other room images.

[0029] Figure 2 A top view of room 100 is shown, where the estimation of the room's principal coordinate axis 202 can be based on lines 208 in an image of seat 204, which may be rotated relative to the room's wall / floor boundaries. Therefore, depending on the location of the captured image (and the objects in the field of view), different (incorrect in this case) coordinate axis estimates may be generated.

[0030] Figure 3A A top view of room 100 with overlaid capture path 302 according to certain exemplary embodiments of the present disclosure is shown.

[0031] Figure 3B It shows Figure 3AThe diagram shows a top view of room 100, where samples collected from within the room along capture path 302 have local coordinate axes 304, 306, 308, 310, 312, 314, 316, 318, and 320. These local coordinate axes are determined based on their orientation and relevant lines extracted from related images of the object. To reduce the number of samples that need to be stored in memory, a minimum distance between local coordinate axis sample points can be enforced. This effectively sets the maximum number of local coordinate axis estimates that can be collected within a given square foot area. According to certain exemplary embodiments of the present disclosure, this figure illustrates a voting process 330, where a "majority-Takes-All" vote can be performed based on the determined local coordinate axes to estimate the room's main coordinate axis. To determine the winner, each local coordinate axis estimate can be considered sequentially and compared with all other local coordinate axis estimates. The amount of deviation between two local coordinate axes can be determined by the minimum angle required to rotate one axis estimate to the other. It can calculate the total angular deviation between each local coordinate axis estimate and all other local coordinate axis estimates, and select the local coordinate axis estimate with the smallest total deviation as the winner to represent the coordinate axis of the entire space.

[0032] In some implementations, if the user returns along their original capture path 302 and enters within a predetermined distance of the existing determined local coordinate axes, the existing determined local coordinate axes can be updated. Therefore, according to some exemplary implementations of this disclosure, even if the user retraces their path multiple times during the scanning process, the number of determined local coordinate axes can be constrained to a maximum number.

[0033] Figure 4 An example of a computing device 400 configured to implement an inspection platform 416 is shown, which is designed to generate measurements of a physical space and its contained objects. The physical space can be an indoor or outdoor space. The inspection platform 416 can generate measurements based on analyzing digital images of the physical space. As detailed below, these digital images can be acquired during a guided measurement operation, in which the user is prompted to reposition the computing device 400 using digital elements.

[0034] The computing device 400 may include a processor 402, a memory 404, a display 406, a communication module 408, an image sensor 410 (such as a camera), and a sensor suite 412. The various components described above will be described in further detail below. Those skilled in the art will recognize that different combinations of these components may exist depending on the nature of the computing device 400.

[0035] Processor 402 may have general-purpose characteristics similar to a general-purpose processor, or processor 402 may be an application-specific integrated circuit (ASIC) that provides control functions for computing device 400. For example... Figure 4 As shown, for communication purposes, processor 402 may be directly or indirectly coupled to all components of computing device 400.

[0036] Memory 404 may include any suitable type of storage medium, such as static random access memory (SRAM), dynamic random access memory (DRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, or registers. In addition to storing instructions executable by processor 402, memory 404 may also store data generated by processor 402 (e.g., when executing a module checking platform 416). Memory 404 may be an abstract representation of a storage environment. Memory 404 may include an actual memory integrated circuit (also called a "chip").

[0037] According to certain exemplary embodiments of the present disclosure, display 406 can be any mechanism operable to visually convey information to a user. For example, display 406 can be a panel including light-emitting diodes (LEDs), organic LEDs, liquid crystal elements, or electrophoretic elements. In some embodiments, display 406 can be touch-sensitive. Therefore, a user can provide input to inspection platform 416 by interacting with display 406.

[0038] The communication module 408 can manage communication between components of the computing device 400, or it can manage communication with other computing devices (e.g., Figure 1 Communication between server system 108 and other computing devices. Communication module 408 may be a wireless communication circuit designed to establish communication channels with other computing devices. Examples of wireless communication circuits include chips configured for Bluetooth, Wi-Fi, NFC, etc.

[0039] Image sensor 410 can be any electronic sensor capable of detecting and transmitting information to generate a digital image, typically in the form of image data or pixel data. Examples of image sensors include charge-coupled device (CCD) sensors and complementary metal-oxide-semiconductor (CMOS) sensors. Image sensor 410 may be implemented in a camera module (or simply "camera") implemented in computing device 400. In some embodiments, image sensor 410 is one of a plurality of image sensors implemented in computing device 400. For example, image sensor 410 may be included in a front-facing or rear-facing camera of a mobile phone (or smartphone).

[0040] Other sensors may also be installed in the computing device 400. These sensors may be collectively referred to as the “sensor suite” 412 of the computing device 400. For example, the computing device 400 may include motion sensors whose output indicates the overall motion of the computing device 400. Examples of motion sensors include accelerometers and gyroscopes. In some embodiments, the motion sensors are implemented in an inertial measurement unit (IMU) to measure the forces, angular rates, or orientations of the computing device 400. The IMU may implement these functions by using one or more accelerometers, one or more gyroscopes, one or more magnetometers, or any combination thereof. As another example, the computing device 400 may include a proximity sensor whose output indicates the proximity of the computing device 400 to the nearest obstacle within the proximity sensor’s field of view. For example, the proximity sensor may include an emitter capable of emitting infrared (IR) light and a detector capable of detecting reflected IR light returning to the proximity sensor. Such proximity sensors are sometimes referred to as laser imaging, detection, and ranging (LiDAR) sensors. As another example, the computing device 400 may include an ambient light sensor whose output indicates the amount of light in the surrounding environment.

[0041] The computing device 400 may also implement the AR framework 414. The AR framework 414 is normally executed by the operating system of the computing device 400, rather than by any individual computer program running on the computing device 400. The AR framework 414 can integrate: (i) digital images generated by the image sensor 410; and (ii) outputs generated by one or more sensors included in the sensor suite 412 to determine the position of the computing device 400 in 3D space. At a high level, the AR framework 414 can perform motion tracking, scene capture, and scene processing to establish the spatial localization of the computing device 400 in real time. Typically, the computer program running on the computing device 400 can access the AR framework 414 via an application programming interface (API). Therefore, the inspection platform 414 can easily obtain the spatial localization from the AR framework 414 via the API, as detailed below.

[0042] For convenience, inspection platform 416 may be referred to as a computer program residing in memory 404. However, inspection platform 414 may include software, firmware, or hardware implemented in or accessible to computing device 400. According to the embodiments described herein, inspection platform 416 may include processing module 418, coordination module 420, measurement module 422, and graphical user interface (GUI) module 424. Each of these modules may serve as an integrated part of inspection platform 416. Alternatively, these modules may be logically independent of inspection platform 416 but operate in conjunction with it. In summary, these modules enable inspection platform 416 to automatically generate measurements of physical space and its content objects by guiding the user through measurement operations.

[0043] According to certain exemplary embodiments of the present disclosure, processing module 418 can process data obtained by inspection platform 416 into a format suitable for other modules. For example, processing module 418 can apply operations to digital images generated by image sensor 410 for analysis by other modules of inspection platform 416. Thus, processing module 418 can perform speckle removal, noise reduction, or other filtering on images generated by image sensor 410. Additionally or alternatively, processing module 418 can adjust attributes such as contrast, saturation, and gain to improve the output produced by other modules of inspection platform 416.

[0044] Processing module 418 can also process data obtained from sensor suite 412 for analysis by other modules of inspection platform 416. As detailed below, inspection platform 416 can utilize data generated by motion sensors to better understand data generated by image sensor 410. For example, inspection platform 416 can programmatically combine digital images generated by image sensor 410 based on measurements generated by motion sensors to create a panoramic view of the physical space. Furthermore, inspection platform 416 can determine the approximate location of each digital image generated by image sensor 410 based on measurements, and then use these insights to establish the dimensions of the physical space and its contents. Alternatively, as image sensor 410 generates digital images, inspection platform 416 can infer movement of computing device 400 based on measurements. For example, inspection platform 416 can determine the direction and magnitude of movement of computing device 400 based on analyzed measurements. To achieve this, measurements generated by motion sensors can be time-aligned with digital images generated by image sensor 410. The processing module 418 is responsible for ensuring that these data are aligned with each other in time, so that the inspection platform 416 can easily identify the measurement value corresponding to each digital image.

[0045] Coordination module 420 may be responsible for determining and / or cataloging the locations of points of interest. In one exemplary embodiment, a user may be interested in establishing the dimensions of a physical space. The perimeter of the physical space may be defined by junctures. The term "juncture" may refer to any location where a pair of walls are connected, intersecting, or otherwise meet or merge. As used herein, the term "juncture" is intended to encompass corners where walls form acute, obtuse, or acuminate angles (an acuminate angle is defined as an angle greater than 180° but less than 360°). Thus, the teachings of this disclosure can be applied to any structure, regardless of its specific configuration. To "map" the perimeter of the physical space, inspection platform 416 may request the user to position computing device 400 at a location (e.g., near the center of the physical space) and then capture a panoramic view of the physical space by translating computing device 400. Coordination module 420 may be responsible for determining the locations of junctures in the physical space based on the analysis of the panoramic view. As detailed below, this functionality can be achieved by applying a trained model to the panoramic view. The trained model can generate coordinates as output, indicating the location of a confirmed boundary, based on pixel-level verification of the panoramic image. The trained model can produce a series of outputs representing different boundaries in the physical space. Using this series of outputs, the coordination module 420 can "reconstruct" the physical space, thereby establishing its dimensions.

[0046] Measurement module 422 can be used to verify the boundary locations determined by coordination module 420 in order to derive information about the orthographically imaged physical space. For example, measurement module 422 can calculate the dimensions of the physical space (e.g., width defined by a pair of wall boundaries or height defined by the ground-to-wall boundary and the ceiling-to-wall boundary) based on comparing multiple locations. Alternatively, measurement module 422 can use locations to generate a 2D or 3D layout. Thus, measurement module 422 can construct a 2D or 3D model of the physical space based on information obtained by analyzing a single panoramic view. In some embodiments, measurement module 422 is also responsible for cataloging the boundary locations determined by coordination module 420. Therefore, measurement module 422 can store these locations in a data structure associated with the physical space or buildings related to the physical space. Information derived by measurement module 422 (such as dimensions and layout) can also be stored in the data structure. In some embodiments, each location is represented using a real-world positioning-related coordinate system (e.g., a geographic coordinate system such as the Global Positioning System), while in other embodiments, each location is represented using a coordinate system related to the surrounding environment. For example, the location of each boundary can be defined for the location of computing device 400.

[0047] As described above, generating a digital image of the entire physical space (or at least the portion under test) helps ensure the accuracy of the information derived by the measurement module 422. To ensure this objective, the inspection platform 416 can prompt the user to move the computing device in a specific manner as the digital image is generated during the measurement operation. For example, the computing device 400 can be positioned vertically or horizontally, thereby defining a vertical plane. In this scenario, the inspection platform 416 can prompt the user to move the computing device along the vertical plane, for example, according to the shape indicated by the digital elements presented on the display 406.

[0048] According to certain exemplary embodiments of the present disclosure, as the image sensor 410 generates digital images during a measurement operation, these digital images can be presented on the display 406 as a video stream. To prompt a user to move the computing device 400, the GUI module 424 can overlay digital features onto the video stream. At a higher level, the digital features may represent augmented reality components designed to prompt the user to move the computing device 400 in a predetermined manner via real-time feedback.

[0049] As detailed below, the digital feature is responsive to movement along a vertical plane. In some embodiments, movement of the computing device 400 can be inferred based on measurements generated by sensors included in the analysis sensor suite 412. For example, the inspection platform 416 can infer the direction and magnitude of movement based on measurements generated by motion sensors included in the computing device 400. In other embodiments, movement of the computing device 400 can be determined based on spatial information output by the analysis AR framework. As described above, a digital image generated by the image sensor 410 can be provided as input to the AR framework during the measurement operation. Whenever a digital image is provided as input to the AR framework, the AR framework can generate spatial information, including the estimated spatial location of the computing device 400 when the digital image was generated. By analyzing these spatial locations estimated by the AR framework, the inspection platform 416 can determine whether the spatial location of the computing device 400 has changed (and thus whether the appearance of the digital feature should be changed).

[0050] In some exemplary embodiments, the GUI module 424 may also be responsible for generating an interface that can be displayed on the display 406. Various types of information can be presented on these interfaces. For example, information calculated, derived, or otherwise obtained by the coordination module 420 and / or the measurement module 422 may be presented on the interface for display to the user. As another example, visual feedback may be presented on the interface to indicate to the user whether the measurement operation was completed correctly.

[0051] Figure 5An example block diagram of a processing system 500 that can implement at least some of the operations described herein is shown. For example, components of the processing system 500 may be hosted on a computing device including an inspection platform, or components of the processing system 500 may be hosted on a computing device for capturing images of an indoor space.

[0052] Processing system 500 may include a central processing unit (“processor”) 502, main memory 506, non-volatile memory 510, network adapter 512, video display 518, input / output device 520, control device 522 (e.g., keyboard or pointing device), drive unit 524 (including storage medium 526), ​​and signal generation device 530 communicatively connected to bus 516. Bus 516 is abstractly represented in the figure as one or more physical buses or point-to-point connections connected via appropriate bridges, adapters, or controllers. Therefore, bus 516 may include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, HyperTransport or Industry Standard Architecture (ISA) bus, Small Computer System Interface (SCSI) bus, Universal Serial Bus (USB), Inter-Integrated Circuit (I2C) bus, or IEEE 1394 standard bus (also known as “FireWire”).

[0053] Although the main memory 506, non-volatile memory 510, and storage medium 526 are shown as a single medium in the diagram, the terms "machine-readable medium" and "storage medium" should be understood to include single or multiple media (e.g., centralized / distributed databases and / or related caches and servers) storing one or more instruction sets 528. The terms "machine-readable medium" and "storage medium" should also include any medium capable of storing, encoding, or carrying instruction sets executed by the processing system 500.

[0054] Generally, routines executed in implementing embodiments of this disclosure may be implemented as a portion of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively, "computer programs"). These computer programs typically include one or more instructions (e.g., instructions 504, 508, 528) set at different times within various memories and storage devices in a computing device. When processor 502 reads and executes the instructions, it causes processing system 500 to perform operations relating to various aspects of this disclosure.

[0055] Further examples of machine-readable and computer-readable media include: recording media, such as volatile and non-volatile memory 510, removable disks, hard disk drives, and optical discs (e.g., compact disc read-only memory (CD-ROM) and digital versatile disc (DVD)); and transmission media, such as digital-analog communication links.

[0056] Network adapter 512 enables processing system 500 to transfer data with external entities within network 514 via any communication protocol supported by processing system 500 and its external entities. Network adapter 512 may include a network adapter card, wireless network interface card, router, access point, wireless router, switch, multilayer switch, protocol converter, gateway, bridge, bridging router, hub, digital media receiver, repeater, or any combination thereof.

[0057] Figure 6 A flowchart of a method 600 for estimating room coordinate axes according to certain exemplary embodiments of the present disclosure is shown. At block 602, method 600 includes receiving an input command at a mobile computing device to initiate the capture of a visual document of an interior environment. At block 604, method 600 includes communicating with an augmented reality (AR) engine via a camera of the mobile computing device to capture multiple images of an interior environment, wherein the interior environment includes one or more walls and multiple objects, each wall and object having a surface oriented along a corresponding plane. At block 606, method 600 includes determining the local coordinate axis orientation of at least a portion of the walls and objects based on information received from the AR engine. At block 608, method 600 includes estimating the room coordinate axes of the interior environment based on a voting process, wherein the room coordinate axes are determined using a majority-matching local coordinate axis orientation. At block 610, method 600 includes outputting an indication of the room coordinate axes.

[0058] Some embodiments of this disclosure may further include outputting annotations corresponding to the local coordinate axis orientation. In some embodiments, at least a portion of the annotations may be output during capture to provide feedback to the user. In some embodiments, the annotations may be aligned with the local coordinate axis orientation and correspond to the physical position and angle of the object within an indoor environment. In some embodiments, the annotations may be aligned with the local coordinate axis orientation and correspond to the position and angle of the intersection of a physical wall and a physical floor.

[0059] In some implementations, multiple images can be captured as the user moves to different locations within the indoor environment.

[0060] In some implementations, the visual document may include a three-dimensional (3D) mapping of the interior environment.

[0061] Some embodiments of this disclosure may include capturing measurements of the indoor environment based on estimated room coordinate axes. In some exemplary embodiments, the measurements may include one or more dimensions and angles.

[0062] According to certain exemplary embodiments of the present disclosure, the indoor environment may include one or more of the following: kitchen, living room, utility room, office, bedroom, bathroom, garage, etc.

[0063] Some embodiments of this disclosure may utilize inertial data generated by one or more position sensors of an AR framework and a mobile computing device to scan and / or map an indoor environment. In some exemplary embodiments, the one or more position sensors may include accelerometers, gyroscopes, and / or similar devices. Some embodiments may include output instructions to prompt a user to move the computing device in a predetermined manner to capture a room scan. Some embodiments may include displaying real-time feedback to prompt the user to adjust the movement of the computing device as digital images are captured. In some exemplary embodiments, the computing device is a smartphone or tablet computer. In some embodiments, the computing device may include a drone or something associated with a drone.

[0064] In some exemplary implementations, the captured spatial information may include spatial coordinates, each spatial coordinate indicating the spatial location of the computing device when the corresponding digital image is captured, and the AR framework generates a corresponding spatial location as output for each captured digital image.

[0065] In some exemplary embodiments, the AR framework may be provided by a commercially available AR engine executable on a computing device, and may use the computing device's camera, processor, and motion / position sensors to perform visual inertial routes to track the surrounding environment and / or perceive how the computing device moves in space. Examples of existing AR frameworks that can be used in conjunction with the technologies disclosed herein can be found in the Apple Developer ARKit documentation (https: / / developer.apple.com / documentation / arkit / ) or the Google ARCore documentation (https: / / developers.google.com / ar / develop), both of which are incorporated herein by reference in their entirety.

[0066] Because augmented reality (AR) engines are widely used in modern mobile devices, certain embodiments of this disclosure can utilize an AR engine to automatically capture the relative position and orientation (i.e., pose) of the camera in a world coordinate system when capturing each image. In cases where an AR system is unavailable, any number of "point-based relative pose" techniques can be used to obtain the relative pose of the camera. However, for certain embodiments of this disclosure, the world coordinate capture position and orientation of the image are known or deducible. Furthermore, according to certain exemplary embodiments of this disclosure, the AR engine can generate a positioning estimate, which can be used, for example, to provide specific feedback to a user and / or determine the location of objects and features within the room during imaging / scanning of a room.

[0067] This disclosure includes methods that can be implemented via computer program instructions executable on a computing device. For example, these instructions may cause the computing device to receive input indicating a request to establish the structural dimensions of an interior space via a scanning process. Such input may correspond to a user initiating (i.e., starting) a computer program or interacting with the computer program in a manner that initiates a measurement of a structure. In response to receiving input initiating a measurement, the computer program may invoke an AR framework executable by the computing device.

[0068] The AR framework can be executed "in the background" by the computing device's operating system, and therefore not by the computer program itself. Instead, the computer program can retrieve spatial information from the AR framework when necessary. For example, the ARKit's ARWorldTrackingConfiguration class can be called to track the six degrees of freedom of the computing device's movement: three rotational axes (roll, pitch, and yaw) and three translational axes (movement in the x / y / z directions).

[0069] ARKit's ARPositionalTrackingConfiguration class enables six degrees of freedom tracking of a computing device by running a camera at the lowest resolution and frame rate. This device tracking information is available both to a computer program executing on the computing device and to the present disclosure for detecting the positioning of the computing device (and its associated camera) as the camera captures images. As a form of guided real-time feedback, this device tracking and / or positioning information can be used to select and / or change the overlay appearance on the computing device's display to instruct the user to move the computing device / camera in a specific pattern so that multiple different views of the scene and / or structure can be imaged, for example, providing additional or enhanced information about structures or objects in the digital image. In some embodiments, moving the camera and processing the digital image can provide "synthetic parallax," which can be used to extract depth information about structures or objects in the digital image. In some exemplary embodiments, the enhanced information about structures or objects in the digital image can be used for a variety of different purposes, including but not limited to structure measurement, object measurement, object recognition, object detection, object condition detection, security hazard detection, etc.

[0070] As part of the measurement process, the user may be prompted to position the computing device so that a digital image of the structure can be generated. To measure the structure, the computer program can utilize and combine information derived from analyzing the digital image. See U.S. Patent Application No. 17 / 500,128, filed October 13, 2021, entitled “Generating Measurements of Physical Structures and Environments Through Automated Analysis of Sensor Data,” and U.S. Patent Application No. 20220114298, filed April 14, 2022, both of which are incorporated herein by reference in their entirety. Specifically, the computer program can enable or facilitate structural measurements based on: (i) digital images generated by an image sensor; and (ii) measurements generated by an inertial sensor (also known as a “motion sensor”). As mentioned above, the advantage of digital images is that while they provide a visual representation of the structure, their scale factor is unknown, whereas inertial measurements (also known as “motion measurements”) provide an estimate of the unknown scale factor. Based on the above data, the structural measurements can be estimated.

[0071] Measurement accuracy can be improved by generating digital images of the structure from multiple spatial locations to achieve broader structural coverage. In the example scenario, a first digital image of the structure can be captured from a first spatial location, and a second digital image of the structure can be captured from a second spatial location. Capturing the first and second digital images from different spatial locations allows a computer program to estimate structural measurements from different perspectives. In short, compared to capturing multiple digital images from the same perspective, the first and second digital images provide more information about the structure (and the overall interior space). Capturing digital images from multiple perspectives also enables other features. For example, if the digital images capture more of the structural surfaces, it is easier to create a digital representation of the structure.

[0072] For illustrative purposes, certain embodiments are described within the context of generating measurements of interior space structures. Examples of structures include the floor, ceiling, and walls of an interior space and their contained objects, such as furniture. However, the methods described herein can also be applied to improve digital image coverage of structures within exterior spaces. Generally, the term "Interior Space" can be used to refer to the physical space inside the building of interest. Meanwhile, the term "Exterior Space" can be used to refer to the physical space outside the building of interest. Examples of exterior spaces include driveways, platforms, etc.

[0073] For illustrative purposes, some embodiments described herein may be implemented within the context of executable instructions. However, those skilled in the art will recognize that various aspects of this disclosure can be implemented in hardware, firmware, or software. For example, a computer program representing a software implementation of an inspection platform (or simply "inspection platform") may be executed by a processor of a computing device to facilitate imaging and measurement of indoor or outdoor spaces. This computer program may interact directly or indirectly with hardware, firmware, or other software implemented on the computing device.

[0074] In the foregoing, the terms "one embodiment" or "some embodiments" mean that the described features, functions, structures, or characteristics belong to at least one embodiment. Such expressions do not necessarily refer to the same embodiment, nor do they necessarily refer to mutually exclusive alternative embodiments.

[0075] The term "based on" should be interpreted as inclusive rather than exclusive, meaning "including but not limited to". Therefore, the term "based on" is intended to mean "based at least in part on", unless otherwise stated.

[0076] The term "connected / coupled" and its variations are intended to encompass any connection or coupling between two or more elements, whether direct or indirect. A connection or coupling can be physical, logical, or a combination thereof. For example, elements can be electrically or communicatively coupled to each other even without sharing a physical connection.

[0077] The term "module" can refer to software, firmware, hardware, or a combination thereof. A module is typically a functional component that generates one or more outputs based on one or more inputs. A computer program may include or utilize one or more modules. For example, a computer program may utilize multiple modules responsible for performing different tasks, or a computer program may utilize a single module responsible for performing all tasks.

[0078] When the term "or" is used to list multiple items, it is intended to cover all of the following interpretations: any one item in the list, all items in the list, and any combination of items in the list.

[0079] The foregoing description of various embodiments of the subject matter of this disclosure is for illustrative purposes only and is not intended to exhaustively cover the subject matter of this disclosure, nor is it intended to limit the subject matter of this disclosure to the specific implementations thereof. Those skilled in the art will understand various modifications and variations. The embodiments were selected and described to illustrate, as clearly as possible, the principles of the invention and its practical application, thereby enabling those skilled in the art to understand the subject matter claimed in this disclosure, the various embodiments, and the various modifications suitable for a particular intended use.

[0080] While certain embodiments have been described in the detailed description, the technology disclosed herein can be implemented in many ways, regardless of the level of detail in the detailed description. The specific implementation details of each embodiment may differ significantly, but they still fall within the scope of protection covered by this specification. The use of specific terminology in describing certain features or aspects of the embodiments should not imply that the redefinition of that term herein is limited to any particular feature, characteristic, or aspect associated with it. Generally, the terminology used in the appended claims should not be construed as limiting the technology disclosed herein to the specific embodiments, unless such terminology is expressly defined otherwise herein. Accordingly, the actual scope of this disclosure covers not only the embodiments of this disclosure but also all equivalent ways of implementing or practicing these embodiments.

[0081] The language chosen in this specification is primarily for readability and explanatory purposes, and is not intended to limit the subject matter. Therefore, the scope of this disclosure should not be limited to the specific embodiments described above, but rather to the appended claims. Accordingly, the embodiments of this disclosure are for illustrative purposes only and are not limited to the scope of protection defined by the appended claims.

Claims

1. A computer-implemented method for estimating room coordinate axes, comprising: At the mobile computing device, input commands are received to initiate the capture of visual documents of the indoor environment; The mobile computing device communicates with the augmented reality (AR) engine via its camera to capture multiple images of the indoor environment, which includes one or more walls and multiple objects, each wall and object having a surface oriented along a corresponding plane. Based on the information received from the augmented reality engine, the local coordinate axis orientation of at least a portion of the wall and objects is determined; Based on the voting process, the room coordinate axes of the indoor environment are estimated, wherein the room coordinate axes are estimated using a majority-matched local coordinate axis orientation; and Output the indication of the room's coordinate axes.

2. The method according to claim 1, further comprising: Output annotations corresponding to the local coordinate axis orientation.

3. The method according to claim 2, wherein, At least a portion of the aforementioned annotations are output during the capture process to provide feedback to the user.

4. The method according to claim 2, wherein, The annotation is aligned with the local coordinate axis orientation and corresponds to the physical position and angle of the object within the indoor environment.

5. The method according to claim 2, wherein, The annotation is aligned with the local coordinate axis orientation and corresponds to the position and angle at which the physical wall and the physical ground intersect.

6. The method according to claim 1, wherein, Multiple images are captured as the user moves to different locations within the indoor environment.

7. The method according to claim 1, wherein, The visual document includes a three-dimensional (3D) mapping of the interior environment.

8. The method according to claim 1, further comprising: The dimensions of the indoor environment are captured based on the estimated room coordinate axes.

9. The method according to claim 1, wherein, The indoor environment includes one or more of the following: kitchen, living room, utility room, office, bedroom, bathroom, and garage.

10. A system for capturing ambient spatial documents, comprising: Mobile computing devices, including: The camera is configured to capture video. One or more processors communicate with the camera; An augmented reality (AR) engine that communicates with the one or more processors; The first memory is configured to store the captured video; A second memory stores computer code that causes the one or more processors to execute: At the mobile computing device, input commands are received to initiate the capture of visual documents of the indoor environment; The mobile computing device communicates with the augmented reality (AR) engine via its camera to capture multiple images of the indoor environment, which includes one or more walls and multiple objects, each wall and object having a surface oriented along a corresponding plane. Based on the information received from the augmented reality engine, the local coordinate axis orientation of at least a portion of the wall and objects is determined; Based on the voting process, the room coordinate axes of the indoor environment are estimated, wherein the room coordinate axes are estimated using a majority-matched local coordinate axis orientation; and Output the indication of the room's coordinate axes.

11. The system according to claim 10, wherein, The computer code also causes the one or more processors to output annotations corresponding to the local coordinate axis orientation for display on the mobile computing device.

12. The system according to claim 11, wherein, At least a portion of the aforementioned annotations are output during the capture process to provide feedback to the user.

13. The system according to claim 11, wherein, The annotation is aligned with the local coordinate axis orientation and corresponds to the physical position and angle of the object within the indoor environment.

14. The system according to claim 11, wherein, The annotation is aligned with the local coordinate axis orientation and corresponds to the position and angle at which the physical wall and the physical ground intersect.

15. The system according to claim 10, wherein, Multiple images are captured as the user moves to different locations within the indoor environment.

16. The system according to claim 10, wherein, The visual document includes a three-dimensional (3D) mapping of the interior environment.

17. The system according to claim 10, wherein, The computer code also enables the one or more processors to capture the dimensions of the indoor environment based on the estimated room coordinate axes.

18. A non-transient computer-readable medium storing instructions thereon, which, when executed by a processor of a computing device, cause the computing device to perform the following operations: At the mobile computing device, input commands are received to initiate the capture of visual documents of the indoor environment; The mobile computing device communicates with the augmented reality (AR) engine via its camera to capture multiple images of the indoor environment, which includes one or more walls and multiple objects, each wall and object having a surface oriented along a corresponding plane. Based on the information received from the augmented reality engine, the local coordinate axis orientation of at least a portion of the wall and objects is determined; Based on the voting process, the room coordinate axes of the indoor environment are estimated, wherein the room coordinate axes are estimated using a majority-matched local coordinate axis orientation; and Output the indication of the room's coordinate axes.

19. The non-transient computer-readable storage medium according to claim 18, wherein, The instructions also cause the computing device to output annotations corresponding to the local coordinate axis orientation for display, wherein, as multiple images are captured as the user moves to different locations in the indoor environment, at least a portion of the annotations are output during the capture to provide feedback to the user.

20. The non-transient computer-readable storage medium according to claim 18, wherein, The annotation is aligned with the local coordinate axis orientation and corresponds to the physical position and angle of the object in the indoor environment or the position and angle where the physical wall intersects the physical ground.

Images