Floor plan extraction

Abstract

The method comprises the following steps: filling missing information by applying a building component technology, and extracting a ring representing a floor plane graph from imperfect or incomplete floor plane graph scanning; during all stages of assembly, the floor plan is displayed to the user (910) to allow the user to make targeted indicia (905) on the display, thereby changing the way of interpretation of incomplete scans. The floor plan (910) is regenerated whenever the indicia (905) are placed.

Description

Technical Field

[0001] The present disclosure relates to creating floor plans and models of buildings; more particularly, but not exclusively, the present disclosure relates to creating connected floor plans by using imperfect floor plan scans. Background Art

[0002] People have long attempted to create floor plans using more automated techniques than using tape measures and graph paper. Depth detection and ranging have been used to create scanning systems that create point clouds of various surfaces within a building. However, converting a point cloud into an understandable floor plan is a very complex problem. The concept of a floor plan seems intuitive, but it is difficult to define rigorously. For example, a definition such as "the area that is walkable after all furniture is removed" is not sufficient because some structures that are not part of the floor plan, such as cabinets, fireplaces, steps, etc., will be included entirely. To illustrate this difficulty, in Figure 1 A vertical slice of the 3-D geometry generated from the LiDAR depth scanner is shown at 100 in FIG. 1. Note that some lines are walls, such as 105, while others are noise, such as 110. In this slice, a person might be able to infer the lines that make up the floor plan, but doing so would implicitly use existing knowledge of how buildings are constructed. This experience helps our brains ignore superfluous information and interpolate missing details. Furthermore, the decision of what a floor plan is is not based solely on geometry; rather, it includes an understanding of how the space is used (its purpose) and what people consider to be important at the time. Floor plans are often defined based on purpose or aesthetics rather than geometry. For example, human input may be required to determine whether adjacent areas are considered separate rooms. As another example, is a fireplace part of the outline of a floor plan? It may be an essential component of the structure, but someone might also draw a floor plan without a fireplace. For these and other reasons, while humans who know what a floor plan should look like can seemingly draw in the missing blanks, such as Figure 2 200 in , but this is very difficult work for computers. Previous implementations have been slow and laborious, allowing for complex modeling that can require a lot of computer time and still require human intervention. These processes also require the mastery of complex technical steps that can take years to master. Being able to automatically generate floor plans in a short period of time, with intuitive user input when needed, would be beneficial. Summary of the invention

[0003] A brief overview of various embodiments is presented below. Some simplifications and omissions may be made in the following brief description, which is intended to emphasize and introduce some aspects of the various embodiments, but not to limit the scope of the embodiments described herein. Detailed descriptions of various embodiments will follow in the following sections, which are sufficient to allow those of ordinary skill in the art to make and use the concepts of the present invention.

[0004] The embodiments include a method for extracting a floor plan from an imperfect floor plan scan consisting of known line segments, comprising: generating a more optimized floor plan using a processor; displaying the more optimized floor plan by a user device having a 3-D depth scanner and a 2-D camera; receiving feedback indicating the accuracy of the more optimized floor plan using the 3-D depth scanner and the 2-D camera; and in response to the feedback, adjusting the more optimized floor plan based on the feedback using the processor.

[0005] The method may include receiving feedback indicating the accuracy of the floor plan, including receiving wall markings from readings from the 3-D depth scanner and the 2-D camera, the wall markings including multiple positions and multiple orientations of locations in 2-D and 3-D, the wall markings indicated by a user on a user device.

[0006] The method may also include processing the plurality of positions and the plurality of orientations of locations in 2-D and 3-D to determine a directed line segment and adding the directed line segment to the known line segments.

[0007] The method may include a processor processing multiple positions and multiple orientations of locations in 2-D and 3-D to determine a directed line segment, and also including determining an average of the multiple positions and multiple orientations using both depth scanner measurements and camera measurements.

[0008] The method may include: for each line segment in the known line segments, extending the line segment by a known amount to create an extended line segment with the same orientation associated with the line segment, and creating a second line segment of a second dimension perpendicular to the line segment and touching the line segment; adding the extended line segment and the second line segment to a set of extended line segments; for each two extended line segments within the set of extended line segments, adding their intersections with similar line segment orientations to a set of intersections, and finding possible edge paths between the two line segments associated with the extended line segment; creating a cost graph including a vertex set and an edge set, wherein the vertex set includes vertices representing the midpoints of each line segment, and wherein the edge set includes edges, each edge including an intersection within the set of possible intersections, a starting vertex of the intersection, an ending vertex of the intersection, and a cost, wherein the cost includes a length from the starting vertex to the ending vertex and an aesthetic cost; and iteratively: in the cost graph, selecting an edge with the lowest cost, giving a selected edge; adding the selected edge to a floor plan table; and deleting each edge whose starting vertex is equal to the starting vertex of the selected edge; until the floor plan table forms a ring.

[0009] The method may include adding the selected edge to the floor plan graph, including adding a possible edge path between two line segments to the floor plan graph.

[0010] A method for finding a possible ending path may include: determining a first angle between two closest ends of two line segments; determining a second angle between a distal end of a first line segment of the two line segments and a distal end of a second line segment of the two line segments; and connecting the first line segment of the two line segments and the second line segment of the two line segments when the first angle is below a first threshold amount and the second angle is below a second threshold amount.

[0011] The method may include zigzagging the two line segments when the first angle is above a first threshold amount or the second line segment is above a second threshold amount.

[0012] The method may include creating a new line segment perpendicular to and between a first line segment of the two line segments while zigzagging the two line segments, and extending proximal ends of the two line segments to the new line segment.

[0013] The method may include adding a weighted value to the cost.

[0014] The embodiments also include a system for extracting a floor plan from an imperfect floor plan scan consisting of line segments, the system comprising: a memory; a processor in operative communication with the memory, and a LiDAR-enabled user device having a display, the processor being configured to extract a floor plan from an imperfect floor plan scan consisting of known line segments, including: generating a more optimized floor plan by applying a heuristic to the imperfect floor plan scan; displaying the more optimized floor plan by the LiDAR-enabled user device; receiving feedback from a user using the LiDAR-enabled user device, the feedback indicating the accuracy of the floor plan; and in response to the feedback, adjusting the more optimized floor plan based on the feedback.

[0015] A line segment in the system may include an orientation, and a known line segment may include a line segment.

[0016] Applying the heuristics for imperfect floor plan scanning in the system may include: for each line segment in the known line segments, extending the line segment by a known amount to create an extended line segment associated with the line segment having the same orientation, and creating a second line segment of a second dimension perpendicular to the line segment and touching the line segment; adding the extended line segment and the second line segment to a set of extended line segments; for every two extended line segments in the set of extended line segments, adding possible intersections with similar orientations of the line segments associated with the extended line segments to a set of possible intersections, and finding possible edges between the two line segments associated with the extended line segments. path; creating a cost graph comprising a vertex set and an edge set, wherein the vertex set comprises a vertex for representing a midpoint of each line segment, and wherein the edge set comprises edges, each edge comprising an intersection in the set of possible intersections, a starting vertex of the intersection, an ending vertex of the intersection, and a cost, wherein the cost comprises a length from the starting vertex to the ending vertex and an aesthetic cost; and iteratively: in the cost graph, selecting an edge with a minimum cost to give a selected edge; adding the selected edge to a floor plan graph; and deleting each edge whose starting vertex is equal to the starting vertex of the selected edge; until the floor plan graph forms a ring.

[0017] Receiving feedback from a user using a LiDAR-enabled user device indicating the accuracy of a floor plan in the system includes displaying a portion of a structure corresponding to an imperfect floor plan scan and receiving purposeful markings made on the display.

[0018] The processor may process the purposeful markings to create a directed line segment, and may add the directed line segment to the known line segments.

[0019] Adjusting, by the processor, the optimized floor plan based on a display may involve regenerating the optimized floor plan.

[0020] The embodiments also include a non-transitory machine-readable storage medium configured with data and instructions that, when executed by at least one processor, cause one or more devices to perform a method for extracting a floor plan from an imperfect floor plan scan consisting of known line segments, the method comprising: using a processor to generate an optimized floor plan by applying a heuristic method to the imperfect floor plan scan; displaying the optimized floor plan through a LiDAR-enabled user device; receiving feedback indicating the accuracy of implementing the floor plan through the LiDAR-enabled user device; and in response to implementing the feedback, using a processor to adjust the optimized floor plan based on the feedback.

[0021] An embodiment of the non-transitory storage medium further comprises: for each line segment in the known line segments, extending the line segment by a known amount to create an extended line segment associated with the line segment having the same orientation, and creating a second line segment of a second dimension perpendicular to the line segment and touching the line segment; adding the extended line segment and the second line segment to a set of extended line segments; for every two extended line segments in the set of extended line segments, adding possible intersections with similar line segment orientations associated with the extended line segment to a set of possible intersections, and finding possible edge paths between the two line segments associated with the extended line segments; creating a path that includes A cost graph comprising a vertex set and an edge set, wherein the vertex set comprises a vertex for representing a midpoint of each line segment, and wherein the edge set comprises edges, each edge comprising an intersection in the set of possible intersections, a starting vertex of the intersection, an ending vertex of the intersection, and a cost, wherein the cost comprises a length from the starting vertex to the ending vertex and an aesthetic cost; and iteratively: in the cost graph, selecting an edge with a lowest cost to give a selected edge; adding the selected edge to a floor plan graph; and deleting each edge whose starting vertex is equal to the starting vertex of the selected edge; until the floor plan graph forms a ring.

[0022] An embodiment of the non-transitory storage medium also includes a LiDAR-enabled user device that also includes a display, and receiving feedback from a user using the LiDAR-enabled user device indicating the accuracy of a floor plan includes the display showing a portion of a structure corresponding to an imperfect floor plan scan and receiving purposeful markings made on the display.

[0023] Embodiments of the non-transitory storage medium also include an optimized floor plan for modifying a device orientation at an orientation represented by the imperfect floor plan scan.

[0024] These and other aspects of the present invention may be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. The following description, while indicating various embodiments and many specific details thereof, is given by way of illustration only and not limitation. Many substitutions, modifications, additions, or reconfigurations may be made within the scope of the embodiments, and the embodiments include all such substitutions, modifications, additions, or reconfigurations. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Non-limiting and non-exhaustive embodiments of the present embodiments are described with reference to the following drawings, in which:

[0026] Figure 1 An example of an imperfect floor plan scan is shown;

[0027] Figure 2 Shows Figure 1 An example of an imperfect floor plan scan of which a more accurate floor plan has been extracted;

[0028] Figure 3 A general example of a suitable computing environment 300 is shown in which the described embodiments may be implemented;

[0029] Figure 4A shows an example of a back view of an example camera and depth detection device that may be used in some of the described embodiments;

[0030] Figure 4B shows an example of a front view of an example camera and depth detection device that may be used in some of the described embodiments;

[0031] Figure 5 An example of a 2-D line segment with position and direction / orientation is shown;

[0032] Figure 6 An example of an imperfect plane scan consisting of 2-dimensional line segments is shown;

[0033] Figure 7 An example of a method of extracting an optimized floor plan from an imperfect floor plan according to some of the described embodiments is shown;

[0034] Figure 8 An example of a display that may be found on a display screen of a depth scanner / camera for showing an optimized floor plan is shown;

[0035] Fig. 9 An example of a display that may be found on a display screen of a depth scanner / camera for showing an optimized floor plan is shown;

[0036] Fig. 10A and 10BAn example of a flow chart describing receiving feedback and adjusting an optimized floor plan based on the feedback is shown;

[0037] Fig.11 shows examples of known line segments that can be connected to form a graphical form of a floor plan;

[0038] Fig.12 shows that some amount of Fig.11 An example of a line segment of ;

[0039] Fig.13 An example of a vertical line drawn between two line segments is shown at 1300;

[0040] Fig.14 Shown by Fig.11 An example of a graph of a set of possible intersections of line segments;

[0041] Fig.15 A graph for describing exponential time is shown;

[0042] Fig.16 Shown in Fig.14 An example where the edge with the lowest cost is already connected;

[0043] Fig.17 shows the second segment intersection group after it has been selected. Fig.16 The vertex-edge graph of ;

[0044] Fig.18 Shows Fig.14 Cost map shown in , which shows the outline of the optimized floor plan.

[0045] Fig.19 An example of a flowchart for describing the execution of connection is shown.

[0046] Figure 20-22 An example in which connection is performed using a straight line is shown.

[0047] Fig.23 and 24 An example of performing a connection using zigzaging is shown;

[0048] Fig.25 An example of a floor plan that fails to form a ring is shown;

[0049] Fig.26 An example of a portion of a floor plan that fails to form a loop is shown;

[0050] Fig. 27 shows examples of intersections of floor plan portions that fail to form a loop;

[0051] Fig.28 An example of two possible connected segments is shown;

[0052] Fig.29 An example of two segments that may not be connected is shown. DETAILED DESCRIPTION

[0053] Representative embodiments of the methods, non-transitory machine-readable media, and systems are disclosed below, and are particularly useful for systems and methods for creating better floor plans from imperfect floor plan scans. The embodiments implement one or more of the techniques described.

[0054] In the following description, many specific details are set forth in order to provide a comprehensive understanding of the current embodiment. However, it is obvious to those of ordinary skill in the art that the specific details used to practice the current embodiment are not necessarily adopted. In other cases, well-known materials or methods are not described in detail to avoid blurring the current embodiment. "One embodiment", "embodiment", "one example" or "example" means that a specific feature, structure or characteristic described in relation to an embodiment or example is included in at least one embodiment of the current embodiment. Therefore, the phrases "in one embodiment", "in an embodiment", "one example" or "example" that appear in various places throughout this specification do not necessarily all refer to the same embodiment or example.

[0055] The systems, devices, and methods described herein may be modified, added to, or omitted without departing from the scope of the present disclosure. For example, the components of the systems and devices may be integrated or separated. In addition, the operations of the systems and devices disclosed herein may be performed by more, fewer, or other elements, and the methods described may include more, fewer, or other steps.

[0056] Additionally, the steps may be performed in any suitable order.

[0057] For convenience, the present disclosure may be described using relative terms, including, for example, left, right, top, bottom, front, back, upper, lower, up and down and other terms. It will be understood that these terms are used for illustrative purposes only and are not meant to be limiting in any way.

[0058] Furthermore, it should be understood that the drawings provided herein are for explanation purposes to persons of ordinary skill in the art and that the drawings are not necessarily drawn to scale. To assist the Patent Office and readers of any patents issued based on this application in interpreting the claims appended hereto, applicants wish to point out that they do not intend for any appended claim or claim element to invoke 35 U.S.C. 112(f) unless the phrase "means for" or "step for" is expressly used in a particular claim.

[0059] Embodiments according to the embodiments herein may be implemented as apparatus, methods or computer program products. Thus, the present embodiments may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects (which may be referred to as a "system"). In addition, the present embodiments may take the form of a computer program product embodied in any tangible medium of expression that carries computer-usable program code.

[0060] Any combination of one or more machine-usable or machine-readable media may be used. For example, the machine-readable medium may include one or more of a portable computer disk, a hard disk, a random access memory (RAM) device, a read-only memory (ROM) device, an erasable programmable read-only memory (EPROM or flash memory) device, a portable compact disk read-only memory (CDROM), an optical storage device, or a magnetic storage device. The computer program code for performing the operations of the current embodiment may be written in any combination of one or more programming languages.

[0061] The flow charts and block diagrams in the flow charts show the architecture, functions and operations of possible implementation schemes of the system, method and computer program product according to various embodiments of the present embodiment. In this regard, each box in the flow chart or block diagram can represent a module, segment or part of a code, which includes one or more executable instructions for implementing one or more specified logical functions. It will also be noted that the combination of each box in the block diagram and / or the flow chart schematic diagram and each box in the block diagram and / or the flow chart schematic diagram can be implemented by a hardware-based special system or a combination of special hardware and computer instructions that perform a specified function or action. These computer program instructions can also be stored in a machine-readable medium, which can instruct a computer or other programmable data processing device to operate in a specific manner so that the instructions stored in the machine-readable medium produce a manufactured product containing an instruction device, which is used to implement the function / action specified in one or more blocks of the flow chart and / or the block diagram.

[0062] As used herein, the terms "comprises," "including," "comprising," "containing," "having," "having" or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to the process, article, or apparatus.

[0063] In addition, unless explicitly stated to the contrary, "or" refers to an inclusive or rather than an exclusive or. For example, condition A or B is satisfied by any of the following conditions: A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist). "Program" is used broadly in this article to include applications, kernels, drivers, interrupt handlers, firmware, state machines, libraries and other code written and / or automatically generated by programmers (also referred to as developers). "Optimization" means improvement, but not necessarily perfection. For example, it may be possible to make further improvements to an already optimized program or algorithm.

[0064] In addition, any examples or descriptions given herein should not be considered as limitations or express definitions of any one or more terms used with them. On the contrary, these examples or descriptions should be considered as describing about a specific embodiment and are illustrative only. It will be understood by those of ordinary skill in the art that any one or more terms used with these examples or descriptions will cover other embodiments that may or may not be provided with them or elsewhere in the specification, and all such embodiments are intended to be included within the scope of the one or more terms. Languages ​​specifying such non-limiting examples and descriptions include, but are not limited to: "for example," "such as," "as," and "in one embodiment."

[0065] The technical features of the embodiments described herein will be readily apparent to those of ordinary skill in the art, and will also be readily apparent to a wide range of attentive readers in a variety of ways. Some embodiments address technical activities rooted in computing technology, such as creating floor plans from 3D point clouds using more efficient computing techniques that use less processing power and less computer time. Based on the description provided, other advantages based on the technical features taught in this application will also be readily apparent to the skilled person.

[0066] Overview

[0067] A rough floor plan can be created using floor plan detection such as augmented reality AR / LiDAR (augmented reality / light detection and ranging), which allows a detailed 3-D point cloud to be captured in real time with reasonable accuracy and precision by walking through the building and surveying the area with a camera / depth scanner device. This 3-D point cloud can be automatically constructed to create a 3-D mesh. A set of 2D line segments that summarize the key structure and shape of the mesh can be extracted, as described in patent application US17 / 459,084 filed on September 15, 2021, which is hereby incorporated by reference in its entirety for all purposes. References Figure 1100 in 100, even the best LiDAR representations often contain small errors. According to various embodiments, a 3-D point cloud or 3-D mesh of an environment is captured using a depth measurement system (e.g., LiDAR) while the same environment is captured in a 2-D artificial reality (AR) environment. Various embodiments then use a combination of user input and machine learning to place annotations on the AR environment. These annotations are placed using 2-D coordinates from the 3-D LiDAR system. View Figure 1 , it might seem that the nearest neighbor lines might just intersect, but this problem is likely to be more computationally challenging. The goal is to draw a set of paths that describe the outline of each room. To do this, a path must be determined through the graph that visits each vertex exactly once. However, this is known as the Hamiltonian path problem and is NP-complete. A closely related problem that might be used to determine floor plans is the "longest path in a graph problem". While this approach avoids shortcuts and theoretically takes less time, it is still NP-hard. Therefore, neither approach will be able to create a connected floor plan in any kind of reasonable time that could be used.

[0068] Since theoretical results are currently not practical to calculate, various embodiments disclosed herein describe systems and methods for improving scans of imperfect floor plans. The floor plan is constructed wall by wall using a user-held LiDAR device with a display screen. The user scans the room and makes marks on the display screen showing the wall locations. The user can view the current floor plan as new walls are determined by user marks, LiDAR calculations, and heuristics. When the floor plan is displayed incorrectly, the user can place another mark to more accurately specify the wall location. This makes it possible to create floor plans that previously would have required too much computer power and too much time to be effectively calculated. There is another solution to this problem in which the user places pins in each corner of the room, but the pin annotations must be placed in sequence as the person walks around the room. This is like trying to wrap a rope or ribbon around a room; if a pin is missed, the process must be started from the beginning.

[0069] A clear advantage of our approach is that users can place markers in any order. If they receive feedback that the previous placement was not sufficient, they can simply add more. This is more interactive, whereas "dropping pins" requires considerable prior planning.

[0070] Figure 3 A generalized example of a suitable machine environment 300 is shown in which the described embodiments may be implemented. Machine environment 300 is not intended to suggest any limitation as to the scope of use or functionality of the disclosure, as the disclosure may be implemented in a variety of general-purpose or special-purpose machine environments.

[0071] refer to Figure 3 , core processing is represented by core processing block 330. Machine environment 300 includes at least one central processing unit 310, GPU 315, and memory 320. Central processing unit 310 executes machine executable instructions and can be a real or virtual processor. Memory 320 can be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. Memory 320 stores software 385 that implements the method of extracting a floor plan from an imperfect floor plan scan.

[0072] The machine environment may have additional functionality. For example, the machine environment 300 includes storage 340 and communication connections 370, which may include one or more input / output devices 350, one or more network connections (e.g., wired, wireless, etc.) 360, and other communication connections (not shown). The communication connection 370 may include one or more wireless transceivers for performing wireless communications and / or one or more communication ports for performing wired communications. In an embodiment, the communication device 370 may be configured to receive captured building features using 2D and 3D sensors. The input / output device may include a two-dimensional camera 352. A depth sensor 354 may also be included, such as a LiDAR (light detection and ranging) system, and a touch screen 356 that allows a user to input information, display graphical information, etc. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the machine environment 300. Typically, operating system software (not shown) provides an operating environment for other software executed in the machine environment 300 and coordinates the activities of the components of the machine environment 300. The computing system within the machine environment may also be distributed; running parts of the software 385 on different CPUs. For example, in some embodiments, some portions of the software 385 may run on a device co-located with the depth sensor 354 or touch screen 356, while other portions of the software may run on another server or virtual machine (not shown) accessible via the network connection 360.

[0073] The storage device 340 may be removable or non-removable and include a disk, magnetic tape or cassette, CD-ROM, CD-RW, DVD, flash drive, or any other medium that can be used to store information and accessed within the machine environment 300. The storage device 340 stores instructions for the software, such as the region of interest detection software 385, to implement the method of extracting a floor plan from an imperfect floor plan scan.

[0074] One or more input devices 350 may be devices that allow a user or other device to communicate with the machine environment 300, such as a touch input device (e.g., keyboard, camera 352), a depth measurement system 354 (e.g., LiDAR system, microphone, mouse, pen, or trackball), a scanning device, a touch screen 356, or other device that provides input to the machine environment 300. For audio, one or more input devices 350 may be a sound card or similar device that accepts audio input in analog or digital form, or a CD-ROM reader that provides audio samples to the machine environment. One or more output devices 350 may be a touch screen, display, printer, speaker, CD burner, or other device that provides output from the machine environment 300. A compass / magnetometer 390 may be included, which may allow the device to determine what direction to take a picture of using the camera 352 and depth sensor 354.

[0075] One or more communication connections 370 allow communication with another computing / machine entity via a communication medium. The communication medium conveys information such as machine executable instructions, compressed graphics information, or other data in the form of a modulated data signal. The communication connection 370 may include an input / output device 350 and an input / output device that allows a client device to communicate with another device via a network 360. The communication device may include one or more wireless transceivers for performing wireless communications and / or one or more communication ports for performing wired communications. These connections may include a network connection, which may be a wired or wireless network, such as the Internet, an intranet, a LAN, a WAN, a cellular network, or another type of network. It is understood that the network 360 may be a combination of multiple different types of wired or wireless networks. The network 360 may be a distributed network with multiple computers, which may be building controllers working in series. The computing connection 370 may be a portable communication device, such as a wireless handheld device, a mobile phone device, and the like.

[0076] A machine-readable medium is any available, non-transitory, tangible medium that can be accessed in a machine environment. By way of example and not limitation, for machine environment 300, machine-readable media include memory 320, storage device 340, communication media, and any combination of the foregoing. Machine-readable storage medium 365 can be used to store instructions 375 and data 380. The data source can be a computing device, such as a general-purpose hardware platform server configured to receive and transmit information via communication connection 370. Machine environment 300 can be an electrical controller that is directly connected to various resources (e.g., HVAC resources) and has CPU 310, GPU 315, memory 320, input device 350, communication connection 370, and / or other features shown in machine environment 300. Machine environment 300 can be a series of distributed computers. These distributed computers can include a series of connected electrical controllers.

[0077] Although the operations of some disclosed methods are described in a particular sequential order for ease of presentation, it should be understood that this description includes resetting unless the specific language described below requires a specific order. For example, the operations described in sequence can be reset or performed simultaneously. In addition, for simplicity, the accompanying drawings may not show the various ways in which the disclosed methods, devices, and systems can be used in combination with other methods, devices, and systems. In addition, terms such as "determine", "construct", and "identify" are sometimes used in the specification to describe the disclosed technology. These terms are generalizations of the actual operations performed. The actual operations corresponding to these terms will vary depending on the specific implementation, and can be easily discerned by a person of ordinary skill in the art.

[0078] In addition, data generated by any disclosed method can be created, updated, or stored on a tangible machine-readable medium (e.g., a tangible machine-readable medium such as one or more CDs, volatile memory components (e.g., DRAM or SRAM), or non-volatile memory components (e.g., a hard drive)) using a variety of different data structures or formats. Such data can be created or updated on a local computer or over a network (e.g., by a server computer), or stored and accessed in a cloud computing environment.

[0079] Figure 4A 400A and Figure 4B400B in the figure shows an example of the back and front of an example user device with a depth scanner and camera. The systems and methods described herein can be implemented using a mobile phone 400A, another portable computing device, or a computing device with a 2-D camera 405A and a depth scanner 410A. In some embodiments, the system includes a user input device. The user input / output device can be a touch screen 405B. The user input device can be associated with a device different from the device that houses the depth scanner and camera.

[0080] To capture the space, in some embodiments, the user walks around, holding a device with both a camera and a depth scanner (e.g. Figure 4A , 4B ) points at an area within the space from multiple angles. A 3-D cloud scan can be created by using a LiDAR depth scanner. When someone walks through a building and uses a 2-D / 3-D scanner to survey the area, augmented reality AR / LiDAR (Augmented Reality / Light Detection and Ranging) allows for the capture of a detailed 3-D point cloud with reasonable accuracy and precision in real time. The 3-D point cloud can be automatically constructed to a certain extent to create a 3-D mesh. However, a representative LiDAR mesh looks like an unstructured "triangle soup" with a lot of noise and artifacts, making it difficult to discern important aspects of the scene, such as distinguishing walls from furniture, appliances, and other materials present. In addition, the amount of raw details makes it difficult to extract areas of interest (such as walls, windows, doors, etc.). For example, buildings contain many non-wall, flat, "wall-like" surfaces, such as cabinets, bookshelves, desks, etc., making it difficult to determine geometric criteria to distinguish between walls and non-wall objects. The definition of these features depends largely on the background.

[0081] Figure 5500 in FIG. 5 shows an example of a 2-D line segment 505 with an orientation (direction) 510, which is used to create an optimized floor plan. Here, the orientation is at one end, but the orientation can be anywhere along the line segment. One way to create an imperfect floor plan scan is to capture a 3-D environment while intelligently detecting features that are not part of the floor plan. A 3-D point cloud or 3-D grid of an environment can be captured using a depth measurement system (e.g., LiDAR) while capturing the same environment in a 2-D artificial reality (AR) environment. The two-dimensional capture from the camera presents a representation of 3-D depth geometry by using a two-dimensional transformation of a three-dimensional grid. The transformation can be provided by software associated with a camera, a depth scanner, or both. The scan includes relevant readings from the sensor, including orientation and compass direction. The user can place an annotation on the display in an area indicating a wall, which is 2-D. Then, the annotations made by the user can be placed in a 3-D grid environment using a pre-trained machine learning algorithm. The annotation is used as a prompt to identify the approximate orientation of the feature that the user is concerned about, for example, in this case, the wall. The placement of the annotations may not indicate exactly where the features are located; instead, they may suggest a general area to look for in the 3-D mesh to find the features of interest. Once the 3-D mesh is complete, the 2-D annotations are converted to specific locations in 3-D space, positioning the 2-D annotations and orientations in both the real-time preview and post-processing steps. The conversion may include, in part, averaging the orientations in 3-D space, averaging the orientations in 2-D space, or a combination of both. In some cases, the placement is delayed until post-processing, which allows for significant improvements in accuracy and consistency. The area around the 3-D scan annotation (which may be a mesh face within the 3-D mesh) is checked for mesh faces with similar orientations to the mesh face at the 3-D scan annotation. These mesh faces may be adjacent mesh faces. When adjacent mesh faces with similar orientations are found, faces adjacent to the found mesh face can be checked, and so on. These mesh faces with similar orientations are then marked as part of the area of ​​interest. Once a portion of the area of ​​interest has been found, it can be determined that this is a wall floor plan portion. The wall floor plan portion can be flattened into a 2-D mark 500 by flattening the mark so that the flattened portion is parallel to the ground. Imperfect floor plans can be calculated by compass and other orientations automatically determined by 3-D scans created by 2-D cameras and LiDAR, for example Figure 1 The imperfect floor plan shown in .

[0082] Such a system for creating imperfect floor plans and location annotations is disclosed in U.S. patent application US17 / 459,084 filed on September 15, 2021, which is incorporated herein by reference in its entirety for all purposes. Figure 6600 shows an example of a graphical representation of a set of known two-dimensional segments 605 having positions and orientations 610 that can describe an imperfect floor plan such as used herein. The imperfect floor plan contains gaps 615 that should be closed to create a more usable floor plan.

[0083] Example

[0084] refer to Figure 7 , flowchart 700 describes a method that may be used by a device in embodiments disclosed herein. The operations of method 700 presented below are intended to be illustrative. In some embodiments, method 700 may be implemented with one or more additional operations not described and / or without using one or more of the operations discussed. In addition, Figure 7 The order in which the operations of method 700 are illustrated in and described below is not intended to be limiting.

[0085] In some embodiments, the method 700 may be implemented by a program 385 stored in the processor and memory element 330, stored in one or more processing devices (e.g., a digital processor, an analog processor, a microprocessing unit, a digital circuit designed to process information, and / or a mechanism for electronically processing information), such as a core processing element, which includes a central processing unit 310 and a memory element 320. The one or more processing devices may include one or more devices that perform some or all of the operations of the methods disclosed herein in response to instructions stored electronically on an electronic storage medium. The one or more processing devices may include one or more devices configured by hardware, firmware, and / or software that are specifically designed to perform one or more operations of the methods and systems described herein. Operation 705 discloses using a processor to generate an optimized floor plan using the methods and systems disclosed herein. Operation 710 discloses displaying the optimized floor plan. The display may be located on a LiDAR-enabled device (e.g., 400B), on a device 356 associated with a computing environment (which may or may not be a touch screen device), or on a different device.

[0086] Figure 8 A display is disclosed that may be found on a display screen of a depth scanner / camera device or on different types of computers having a display screen. The display may be a background of a portion of the scanned structure, such as seen through a viewfinder, e.g. Figure 8800 . The display may also show a mark 805 made by a user on the display screen, which marks a wall previously used to develop an optimized floor plan 810. For example, a user may drag across the screen with a finger or stylus, leaving a colored mark 805 or other indication of the shape depicted thereby. In some embodiments, the mark 805 may be associated in three-dimensional space with the orientation of the surface on which the mark 805 is drawn. Thereafter, as the orientation of a camera device (e.g., camera 352) changes and the view on the display of the marked surface changes, the image of the mark 805 may be similarly transformed, such as translated, rotated, scaled, tilted, etc., so that the mark 805 remains displayed on the marked surface while the surface remains visible on the display.

[0087] The optimized floor plan may not be the best possible optimization, as shown in the example preview panel 810, where the ongoing floor plan appears to be two triangular rooms, which appears to be an error because three walls of a single room have been scanned at this point (the first two walls and their deliberate markings are not shown). Therefore, ideally, the floor plan should show a single triangle because the fourth wall has not yet been scanned. The two triangles shown indicate that another deliberate marking should be made to provide more information to more accurately determine the floor plan. In an embodiment, when the view on the display 800 is a real-time augmented reality view of the space being mapped, the display can also show the direction in which a particular photo was taken or the direction of the camera device relative to the ongoing floor plan. This is represented by the arc 815; the vertex represents the user's position and the arc represents the camera's field of view. This allows the user to position themselves in the space to determine how accurate the more optimized floor plan display 810 is.

[0088] The processor receives feedback regarding the more optimized floor plan in operation 715. A user viewing the more optimized floor plan (e.g., 810) may notice that the floor plan is incorrect and may therefore make another deliberate mark, e.g., to provide more feedback to the floor plan extraction program. Fig. 9 An example is shown in 900 of FIG. Figure 8, but the depth scanner is moved to show another portion that was not previously in view. The user can then place another deliberate mark 905 on the display. The device (or a processor connected to the device via a wired or wireless connection) can then process the received information to determine a new floor plan line segment, which can then be used to more accurately determine the floor plan. In some embodiments, in order to turn the deliberate marks into line segments, multiple depth measurements captured by the depth sensor can be obtained. These depth measurements can be obtained from a program associated with depth measurement capture. The depth measurement uses a three-dimensional grid to define the distance from the depth sensor to a corresponding point in the space.

[0089] The deliberate marking (which was originally created in two dimensions) can be mapped onto a three-dimensional grid using depth measurements to form three-dimensional rays. The two dimensions can be detected in a two-dimensional grid (associated with the display) and can then be converted to a three-dimensional grid grid orientation using software available using a 3-D grid system, a 2-D camera system, some combination, etc. For the detected points within the deliberate marking, for at least some points, the two dimensions of the point are then converted into a ray with a third-dimensional tail that intersects the three-dimensional shape to which the region of interest is attached. The ray is then tracked to see where it hits the grid in the 3-D world, thereby creating a 3-D ray where the point of the ray intersects the wall. Multiple 3-D rays along the deliberate marking can then be averaged along their ray points to determine a flat 2-D directed line segment. When the 2-D line segment is placed in a manner that the processor can understand (e.g., feedback), at operation 720, the feedback is used to adjust a more optimized floor plan. The optimization 910 can then be shown on the display 900. In some cases, the optimization can be displayed in a determination area 915 within the display. Here, the determination area 915 is demarcated by a dotted line. Other determination areas and demarcations are also within the scope of the present disclosure. When shown, if the more optimized feedback 910 is still incorrect, the user can place another deliberate mark until the floor plan looks correct.

[0090] Fig. 10A 1000A and Fig. 10B 1000B shows an example of a flow chart that describes in more detail receiving feedback and adjusting a more optimized floor plan based on the feedback. In operation 1005, a wall mark is received. Reference Figure 8 and Fig. 9 and surrounding text discuss receiving wall markers. In operation 1010, line segments and orientations are determined. Determining new line segments may include averaging multiple positions and orientations 1015. The line segments represent wall portions in the incomplete portion of the floor plan. The line orientation is associated with the line segment and points to the middle of the assumed room. Reference Figure 5 and Figure 6 and the surrounding text describe these in more detail. In operation 1020, the new line segment is added to the known line segments. Fig.11 1100 graphically illustrates examples of known line segments that can be connected to form a floor plan. A single line segment with an orientation is shown at 1110. This line segment may have been added because line segment 1105, as well as other line segments, may have produced a non-optimal floor plan, such as the floor plan shown at 810. At operation 1025, each line segment is extended by a certain amount, thereby creating an extended line segment. For example, the line segment may be extended by a fixed amount, by an amount within a certain range, different segments may be extended by different amounts, or another method may be used. In some cases, a threshold amount may be set that defines the maximum amount a line may be extended. Fig.12 The 1200 shows Fig.11 At operation 1020, these extended line segments are added to a set of extended line segments.

[0091] This graph of extended segments represents possible paths, but not all possible paths to connect the segments. Sometimes segments are misplaced; for example, the original graph may have multiple segments that should have been connected to form a single segment, but were incorrectly offset. Fig.11 1027, with reference to segments 1105 and 1110. These two offset segments represent the same wall. When these two lines are extended 1205, 1210, the two lines will extend in parallel. Other offset line segments may capture actual surfaces containing pillars, narrow corridors, etc. It is very important to be able to distinguish between errors and actual building features. To this end, in operation 1027, lines perpendicular to the line segments are created. These perpendicular lines can be created for each line segment, each extended line segment, or can be created as a subset of the segments. Perpendicular lines can be created at the end, in the middle, or in some other locations of the line segments. Fig.13 1300 in FIG. 1 shows an example of such a vertical line intersecting another line segment. Line segment 1310 (e.g., line segment 1110, extended line segment 1210) has a vertical line 1315 created at right end 1320. The length of vertical line 1315 can be a fixed length, can depend on the length of line segment 1310, can be a length range, and so on.

[0092] At operation 1030, the intersections of these line segments with the extended segments 1305, 1310 are then found (e.g., 1320, 1325). To do this, a scan line algorithm such as the Shamos-Hoey algorithm, the Bentley-Ottmann algorithm, or a different algorithm may be used. Intersections that cannot be wall intersections due to different orientations are removed. Fig.14An example of a graph 1400 is shown, which is a graph through Fig.11 A set of possible intersections of line segments in . The figure shows an original segment (eg, 1405) and an extended segment (eg, 1410), and a diagram of the extensions (eg, 1410, 1415) intersecting at an intersection (eg, 1420).

[0093] Now the graph 1400 is examined to find a good path through it. Now for each segment an outgoing intersection and an incoming intersection should be selected in order to build a floor plan from the line segments that match the actual floor plan. The question is how to choose which segments will generate the floor plan. To do this, the method finds a path through the graph that visits each vertex (original line segment, e.g., 1405) exactly once. In other words, the method aims to include each actual segment (e.g., Fig.11 ) without any shortcuts. This could produce a cycle through all actual segments. However, finding such a cycle is a well-known and very hard problem to solve, called a Hamiltonian cycle. Determining the existence of a cycle through the segments, even for graphs of restricted degree, is NP-complete. Another heuristic that can be used is "longest in graph", which is also NP-complete. Both theoretical solutions have hard exponential time complexity, since the time to solve doubles for each added element, e.g. Fig.15 As shown in the 1500 charts.

[0094] Various embodiments disclosed herein utilize heuristics to generate solutions in polynomial time. This is accomplished by using the concept of "closeness" between segments. This makes sense from a user experience perspective because users place markers close to each other because they want them to be connected. Both the distance between segments and the angle between selected close segments are indicators that two segments should be connected together. At operation 1035, a cost graph is created. The cost graph may include a vertex set 1040 and an edge set 1045. The vertex set may include existing segments (e.g., Fig.11 , or part or all of the original segments shown. These existing line segments can be represented by two-dimensional points - vertices. The point can be the middle of the line segment, or a different position within the existing line segment. The edge set may include possible intersections / connection relationships (e.g., 1420) between two segments represented in the graph, which uses extended segments and lines perpendicular to the segments to find intersections. The edge set value may include the actual intersection orientation between the two extended segments, and the two vertices of the non-extended segment whose extension produces the intersection. Some embodiments may store the non-extended segment as part of the edge set, and so on.

[0095] At operation 1050, the cost of the edge is found. The cost of an edge can be the length from the first vertex in the edge (e.g., 1405) through the intersection (e.g., 1420) to the second vertex (e.g., 1425). The cost can include a vertical distance multiplied by a constant, a horizontal distance multiplied by a constant, and so on. The cost can be any combination of the above. At operation 1060, the edge with the lowest cost (e.g., the shortest length intersection segment) is determined. At operation 1065, the two segments with the shortest length intersection (e.g., original segments 1115, 1120) are connected. Fig.16 1600 in shows Fig.14 In the connected example, a line segment is created that extends from a point in one of the original line segments represented in the selected edge (in this example, the midpoint 1607) through the intersection 1605 (represented in the selected edge) to a point in the other line segment (represented in the selected edge) 1609. Fig.19 and surrounding text discusses connecting line segments in more detail. In operation 1070, the connected segments are added to the more optimized floor plan. In operation 1075, the edges and corresponding ends of the two vertices to be connected, as well as each edge connected to the connecting end of the vertices (line segments), are deleted from the graph. When a given segment is connected, examples of edges that are deleted are 1610, 1615, and 1620. Fig.17 1700 shows the vertex-edge graph after a second set of segment-intersections 1705 has been selected and the cost graph segments having the same starting position as the edge with the lowest cost have been deleted.

[0096] At decision point 1080, it is determined whether a loop has been created. If it has, then the floor plan is displayed at operation 1090. If not, then at step 1085, if there are no remaining vertices in the cost graph, then the vertices that lead to the dead end are deleted at decision point 1087. Figure 25-29 and surrounding text describe this in more detail. The floor plan is then constructed again starting from operation 1060. In some embodiments, after displaying the floor plan in step 1090, the method can receive additional information from the user, such as, for example, another deliberate 905 addition. In some embodiments, the method can receive modified scan data from the camera 352 or the depth sensor 354 (e.g., rescanning after adding a new changed environment (e.g., by moving furniture or closing a door)), etc. In some such embodiments, the method 1000A, 1000B can recalculate the floor plan by, for example, looping back to step 1005.

[0097] Fig.18 It was shown at 1800 Fig.14Cost graph shown in , which outlines a more optimized floor plan 1805. Fig. 9 At 910, a display of the floor plan after the three walls have been scanned and after it has been further optimized is shown. In operation 1095, the floor plan constructed here is used to make decisions that will change the nature of the space represented by the floor plan and / or change the space in which the floor plan is located. Many variations are possible. Among them, some possible variations can be to determine where sensors can be placed, and then place the sensors in the building; use the floor plan to determine the floor space, use the floor space to determine how much heating and cooling is needed, use the required amount of heating and cooling to determine the size of the HVAC system to be placed in the building, purchase the HVAC system, and then install the HVAC system in the building.

[0098] Fig.19 1900 in the flowchart shows an example, which describes in more detail the connection of one end of two connecting segments (e.g., operation 1065). In some cases, the two lines to be connected are offset from each other by a certain amount. In this case, it can be decided whether to connect these lines by assuming that the two lines should actually be combined into a single line, or whether to connect these lines by forming a zigzag corner. In operation 1905, a first angle between the closest ends of the two line segments to be connected is determined. Fig. 20 2000 in FIG. 2000 shows an example of measuring a first angle 2015 between the nearest ends 2010, 2020 of two line segments (2005-2010 and 2020-2025) to be connected. In operation 1910, a second angle between the farthest ends of the line segments is determined. FIG. 2100 shows an example of measuring a second angle 2115 between the farthest ends 2105, 2125 of two line segments (2105-2110 and 2120-2125) to be connected. In operation 1915, it is determined whether the line segments should be connected by turning them into a single straight line. When the first angle is less than (or less than or equal to) a first threshold value, and when the second angle is less than (or less than or equal to) a second threshold value, they should be connected in this way. The first angle threshold value and the second angle threshold value can have a wide range of values. Fig. 22 2200 in FIG. 2204 shows an example of connecting two points by connecting the farthest point of the first segment 2205 and the farthest point of the second segment 2225 with a single straight line 2230 to turn the two points into a single straight line.

[0099] In operation 1920, the line segment is zigzag. "Zigzag" is related to human knowledge of how buildings are constructed: buildings are usually built at right angles. The difference is likely to be a small joint or column that is not fully described by the individual segments. With this in mind, in some embodiments, zigzag is selected when the first angle (e.g., 2015) is above a threshold or the second angle (e.g., 2115) is above a threshold. Fig.23 As shown in the example given at 2300 in FIG. 1 , the angle of the line segment is above a threshold value 2340. To zigzag the line, the intersection of the two closest endpoints 2310 and 2320 is used to create an xy axis 2335. The xy axes may intersect at a 90 degree angle or other selected angle. Fig.24 As shown in the example given at 2400 in FIG, in the selected dashed axis, the line from the intersection of the xy axis 2435 to one of the proximal endpoints 2420 is divided in half (or divided by some other percentage), and then a zigzag line is drawn between the two proximal points 2410, 2420, with (percent) of the selected dashed axis at one end 2440 and (100 percent) of the drawn selected line at the other end, with a line drawn between them parallel to the non-selected axis 2445. Other methods of drawing a line between two discontinuous points (e.g., 2410, 2420) may also be used.

[0100] In some cases, the floor plan scans did not form a ring. Fig.25 2500 shows an example of a floor plan 2515 and its original segment 2505 and extended segment 2510, which fail to form a loop. Fig.26 2600 and Fig. 27 2700 in FIG. 2 shows two close-ups of segments that are not connected to 2520. Two marks 2605 and 2610 are not connected because, as shown in the close-up of intersection 2700, mark 2705 is slightly on the wrong side of intersection 2710.

[0101] To discuss the concept of what can and cannot be connected, Fig.28 2800 in shows an example of two segments that may be connected. Fig.292900 in shows an example of two segments that cannot be connected. In both figures, the arrows on the segments are on the left relative to the orientation. That is, they represent what is "inside" the room. When the segments may intersect, the intersection 2815 may be in front of one potentially intersecting segment 2810 and behind the other potentially intersecting segment 2805. When both segments are in front of or behind the intersection, they may not intersect. Segments 2905 and 2910 are both behind the intersection 2915. In this case, the orientation is different between the two markers, so they cannot be connected. This can be visualized as the erroneous segment 2905 being outside the room, which is bounded here by segments 2920 and 2910. These segments that can only be connected once result in a dead end. As shown in reference Fig. 27 It can be seen that the wrong orientation just happens - one segment is just outside the range of correct connection. These even slightly misaligned segments occur due to small errors that can accumulate. If you delete this segment that can only be connected on one side, you can connect the correct segment that defines the floor plan, such as 2920. To solve this problem, refer to Fig. 10A and 10B , at operation 1080, the loop of the floor plan graph has not been created, and at 1085 there are no remaining vertices to connect. This indicates that something has gone wrong. At 1087, the vertex that caused the dead end is deleted. The cost graph at this point (e.g., 1035, 1075) mirrors the floor plan graph with the missing vertices. The cost graph is scanned for vertices that are only connected to another vertex at a single end. This will find vertices with a single connection, such as reference Fig. 27 2705 ​​in. In some embodiments, more than one vertex with a single connection may be located. These vertices are then deleted. After deletion, in some embodiments, methods 1000A, 1000B may recalculate the floor plan by, for example, looping back to step 1005, step 1060, or another step.

[0102] Although various example embodiments have been described in detail with particular reference to some of their example aspects, it should be understood that the present invention is capable of other embodiments and that its details can be modified in various obvious aspects. As will be readily appreciated by those skilled in the art, various changes and modifications may be implemented while remaining within the spirit and scope of the present invention. Therefore, the foregoing disclosure, description and drawings are intended for illustrative purposes only and do not limit the present invention in any way, the present invention being limited only by the claims.

Claims

1. A machine-implemented method for extracting a floor plan from an imperfect floor plan scan consisting of known line segments, comprising: Use processors to generate more optimized floor plans; displaying the more optimized floor plan via a user device having a 3-D depth scanner and a 2-D camera; receiving feedback using the 3-D depth scanner and the 2-D camera, the feedback indicating the accuracy of the more optimized floor plan; as well as In response to the feedback, adjusting, using the processor, the more optimized floor plan based on the feedback.

2. The machine-implemented method of claim 1 , wherein receiving feedback indicating the accuracy of the floor plan comprises receiving wall markings from readings from the 3-D depth scanner and the 2-D camera, the wall markings comprising multiple positions and multiple orientations of locations in 2-D and 3-D, the wall markings being indicated by a user on the user device.

3. The machine-implemented method of claim 2, wherein the processor processes the multiple positions and multiple orientations of locations in 2-D and 3-D to determine directed line segments, and wherein the directed line segments are added to the known line segments.

4. The machine-implemented method of claim 3, wherein the processor processes the multiple positions and multiple orientations of the 2-D and 3-D locations to determine directed line segments further comprises determining an average of the multiple positions and multiple orientations using both depth scanner measurements and camera measurements.

5. The machine-implemented method of claim 4, wherein applying a heuristic to the floor plan comprises: for each line segment in the known line segments, extending the line segment by a known amount to create an extended line segment associated with the line segment having the same orientation and to create a second line segment of a second dimension perpendicular to and touching the line segment; adding the extended line segment and the second line segment to a set of extended line segments; For each two extended line segments in the set of extended line segments, adding possible intersections whose line segments have similar orientations to a set of possible intersections, and finding possible edge paths between the two line segments associated with the extended line segments; creating a cost graph comprising a vertex set and an edge set, wherein the vertex set comprises a vertex representing a midpoint of each line segment, and wherein the edge set comprises edges, each edge comprising an intersection in the set of possible intersections, a starting vertex of the intersection, an ending vertex of the intersection, and a cost, wherein the cost comprises a length from the starting vertex to the ending vertex and an aesthetic cost; and iteratively: In the cost graph, selecting an edge having a lowest cost, giving a selected edge; adding the selected edge to a floor plan diagram; as well as Delete each edge whose start vertex is equal to the start vertex of the selected edge; until the floor plan graph forms a ring. 6 . The machine-implemented method of claim 5 , wherein adding the selected edge to the floor plan graph comprises adding a possible edge path between two line segments to the floor plan graph.

7. The machine-implemented method of claim 6, wherein finding a possible ending path comprises: determining a first angle between two closest ends of the two line segments; determining a second angle between a distal end of a first of the two line segments and a distal end of a second of the two line segments; as well as When the first angle is below a first threshold amount and the second angle is below a second threshold amount, a first line segment of the two line segments and a second line segment of the two line segments are connected.

8. The machine-implemented method of claim 7, wherein when the first angle is above the first threshold amount or the second line segment is above the second threshold amount, the two line segments are zigzag.

9. The machine-implemented method of claim 8, wherein meandering the two line segments comprises creating a new line segment perpendicular to and between a first of the two line segments, and extending the proximal ends of the two line segments to the new line segment.

10. The machine-implemented method of claim 9, wherein the cost further comprises adding a weighted value to the cost.

11. A system for extracting a floor plan from an imperfect floor plan scan consisting of line segments, the system comprising: Memory; a processor in operable communication with the memory, and a LiDAR-enabled user device having a display, the processor configured to extract a floor plan from an imperfect floor plan scan consisting of known line segments, comprising: generating a more optimized floor plan by applying a heuristic to the imperfect floor plan scan; displaying, by the LiDAR-enabled user device, the more optimized floor plan; receiving feedback from a user using the LiDAR-enabled user device, the feedback indicating an accuracy of the floor plan; and In response to the feedback, the more optimized floor plan is adjusted based on the feedback.

12. The system of claim 11, wherein a line segment comprises an orientation, and wherein the known line segment comprises a plurality of line segments.

13. The system of claim 12, wherein applying a heuristic to the imperfect floor plan scan comprises: For each line segment in the known line segments, extending the line segment by a known amount, creating an extended line segment associated with the line segment having the same orientation, and creating a second line segment of a second dimension perpendicular to and touching the line segment; adding the extended line segment and the second line segment to a set of extended line segments; For every two extended line segments in the set of extended line segments, adding possible intersections whose line segments associated with the extended line segments have similar orientations to a set of possible intersections, and finding possible edge paths between the two line segments associated with the extended line segments; creating a cost graph comprising a vertex set and an edge set, wherein the vertex set includes a vertex representing a midpoint of each line segment, and wherein the edge set includes edges, each edge including an intersection in the set of possible intersections, a starting vertex of the intersection, an ending vertex of the intersection, and a cost, wherein the cost includes a length from the starting vertex to the ending vertex and an aesthetic cost; and Iteratively: In the cost graph, selecting an edge having a lowest cost, giving a selected edge; adding the selected edge to the floor plan graph; as well as Delete each edge whose start vertex is equal to the start vertex of the selected edge; until the floor plan graph forms a ring.

14. The system of claim 13, wherein receiving feedback from a user using the LiDAR-enabled user device indicating the accuracy of the floor plan comprises displaying a portion of a structure corresponding to the imperfect floor plan scan and receiving purposeful markings made on the display.

15. The system of claim 14, wherein: The processor processes the purposeful markings to create directed line segments and adds the directed line segments to the known line segments.

16. The system of claim 15, wherein: The processor adjusting the optimized floor plan based on a display includes regenerating the optimized floor plan.

17. A non-transitory machine-readable storage medium configured with data and instructions that, when executed by at least one processor, cause one or more devices to perform a method of extracting a floor plan from an imperfect floor plan scan consisting of known line segments, the method comprising: generating, using a processor, a more optimized floor plan by applying a heuristic to the imperfect floor plan scan; displaying the more optimized floor plan via a LiDAR-enabled user device; receiving feedback by using a LiDAR-enabled user device, the feedback indicating an accuracy of the floor plan; and In response to the feedback, a processor is used to adjust the more optimized floor plan based on the feedback.

18. The non-transitory machine-readable storage medium of claim 17, wherein applying a heuristic to the imperfect floor plan scan comprises: for each line segment in the known line segments, extending the line segment by a known amount to create an extended line segment associated with the line segment having the same orientation and to create a second line segment of a second dimension perpendicular to and touching the line segment; adding the extended line segment and the second line segment to a set of extended line segments; For every two extended line segments in the set of extended line segments, adding possible intersections having similar line segment orientations as those associated with the extended line segments to a set of possible intersections, and finding possible edge paths between the two line segments associated with the extended line segments; creating a cost graph, the cost graph comprising a vertex set and an edge set, wherein the vertex set comprises a vertex representing a midpoint of each line segment, wherein the edge set comprises edges, each edge comprising an intersection in the set of possible intersections, a starting vertex of the intersection, an ending vertex of the intersection, and a cost, wherein the cost comprises a length from the starting vertex to the ending vertex and an aesthetic cost; and iteratively: In the cost graph, selecting an edge having a lowest cost, giving a selected edge; adding the selected edge to the floor plan graph; and Delete each edge whose start vertex is equal to the start vertex of the selected edge; Until the floor plan diagram forms a ring.

19. The non-transitory machine-readable storage medium of claim 18, wherein: The LiDAR-enabled user device also includes a display, and wherein receiving feedback from a user using the LiDAR-enabled user device indicating the accuracy of the floor plan includes the display showing a portion of a structure corresponding to the imperfect floor plan scan and receiving a purposeful mark made on the display.

20. The non-transitory machine-readable storage medium of claim 19, wherein: The more optimized floor plan is used to modify the device orientation at the orientation represented by the imperfect floor plan scan.

Images