Floor plan extraction

Abstract

By applying building construction techniques to fill in missing information, a loop shape representing a floor plan is extracted from an incomplete or unfinished floor plan scan. The floor plan (910) is presented to the user at all stages of assembly, thereby enabling the user to make intentional marks (905) on the display and changing the way the incomplete scan is interpreted. Whenever a marking (905) is placed, the floor plan (910) is regenerated.

Description

Technical Field

[0001] The present disclosure relates to the creation of building floor plans and models, and more specifically, the present disclosure relates to, but is not limited to, the creation of connected floor plans using incomplete floor plan scans.

Background Art

[0002] Over the years, attempts have been made to create floor plans using automated techniques different from using a tape measure and graph paper. Scanning systems have been created that use depth detection and ranging to create point clouds of various forms of surfaces within a building. However, turning the point cloud into an understandable floor plan is an extremely complex problem. Although the concept of a floor plan may seem intuitive, it is difficult to define precisely. For example, a definition such as "a walkable area with all furniture removed" is not sufficient because it would take into account all structures that are not part of the floor plan, such as cabinets, fireplaces, and steps. To illustrate such difficulties, a vertical slice of a 3D geometry generated from a LiDar depth scanner is shown at 100 in FIG. 1. Note that some of the lines are walls, such as 105, and other lines are noise, such as 110.

[0003] In this slice, even if it is possible to estimate the lines that generally make up the floor plan, in that estimation, prior knowledge about how the building is constructed is being implicitly used. This experience helps the brain ignore extra information and insert missing details. Also, the determination of what a floor plan is like is not based solely on geometry. It includes an understanding of how the space is used (its purpose) and what a person considers important at that time. The definition of a floor plan is often expressed not in terms of geometry but in terms of use or aesthetics. For example, whether adjacent areas are considered separate rooms may require human input. As another example, will the outline of the fireplace part of the floor plan be drawn? The fireplace part may be a fundamental part of this structure, but there may be people who draw a floor plan without a fireplace. For the above and other reasons, a person who knows what a floor plan should look like, as shown at 200 in FIG. 2, would seem to be able to draw the missing gaps, but this is an extremely difficult task for a computer. Conventional implementations have been time-consuming and labor-intensive in that the complex modeling that can be done takes hours on a computer and may still require human intervention. These processes also require proficiency in complex technical steps that can take years to acquire. It would be advantageous if, when necessary, intuitive user input could be used to automatically generate a floor plan in a short time. SUMMARY OF THE INVENTION

[0004] The following presents an overview of various embodiments. The following overview, which emphasizes and outlines some aspects of the various embodiments and is not intended to limit the scope of the embodiments described herein, may have some simplifications and omissions. A detailed description of the embodiments sufficient to enable those skilled in the art to make and use the concepts of the present invention is described in subsequent sections.

[0005] An embodiment is a method for extracting a floor plan from an incomplete floor plan scan consisting of known line segments, using a processor to generate a more optimized floor plan, displaying a more optimized floor plan by a user device having a 3D depth scanner and a 2D camera, receiving feedback indicating the accuracy of the more optimized floor plan using the 3D depth scanner and the 2D camera, and using a processor to adjust the more optimized floor plan based on the feedback in response to the feedback.

[0006] The method can include receiving wall markings from readings from a 3D depth scanner and a 2D camera, the wall markings including a plurality of positions and a plurality of orientations of locations in 2D and 3D, the wall markings being indicated by a user on the user device.

[0007] The method can also include processing a plurality of positions and a plurality of orientations of locations in 2D and 3D to determine directed line segments, the directed line segments being added to the known line segments. The method can include the processor processing a plurality of positions and a plurality of orientations of locations in 2D and 3D to determine directed line segments, and further including using both depth scanner measurements and camera measurements to determine an average of the plurality of positions and the plurality of orientations.

[0008] The method extends each line segment within a known line segment by a known amount to create an extended line segment associated with the line segments having the same orientation, creates a second line segment of a second size that is perpendicular to and in contact with that line segment, adds the extended line segment and the second line segment to a set of extended line segments, for each pair of extended line segments within the set of extended line segments, adds an intersection where the line segment orientations are similar to a set of intersections, finds a possible edge path between two line segments associated with the extended line segment, and creates a cost graph including a vertex set and an edge set, where the vertex set includes vertices representing the midpoints of each line segment, the edge set includes edges, and each edge includes an intersection within the set of possible intersections, a start vertex of the intersection, an end vertex of the intersection, and a cost, and the cost includes the length from the start vertex to the end vertex and an aesthetic cost, creating a cost graph, and iteratively, until the floor plan graph forms a loop, in the cost graph, selects the edge with the lowest cost, provides the selected edge, adds the selected edge to the floor plan graph, and deletes each edge having a start vertex equal to the selected edge start vertex, and can include.

[0009] The method can include that adding the selected edge to the floor plan graph includes adding a possible edge path between two line segments to the floor plan graph. The method of finding a possible edge path can include determining a first angle between the two closest ends of two line segments, determining a second angle between the far end of the first line segment of the two line segments and the far end of the second line segment of the two line segments, and connecting the first line segment of the two line segments and the second line segment of the two line segments when the first angle is less than a first threshold amount and the second angle is less than a second threshold amount.

[0010] The method can include zigzagging the two line segments when the first angle exceeds a first threshold amount or the second line segment exceeds a second threshold amount. The method can include creating a new line segment that is perpendicular to the first of two line segments and is between the two line segments, and extending the proximal ends of the two line segments to the new line segment when zigzagging the two line segments.

[0011] The method can include a weighting value added to the cost. An embodiment is a system for extracting a floor plan from an incomplete floor plan scan consisting of line segments, including a memory, a processor operably communicating with the memory, and a LiDAR-enabled user device having a display, the processor being configured to extract a floor plan from an incomplete floor plan scan consisting of known line segments, generating a more optimized floor plan by applying heuristics to the incomplete floor plan scan, displaying the more optimized floor plan by the LiDAR-enabled user device, receiving feedback indicating the accuracy of the floor plan from a user using the LiDAR-enabled user device, and adjusting the more optimized floor plan based on the feedback in response to the feedback.

[0012] The line segments in the system can include an orientation, and the known line segments can include line segments. Applying heuristics to an incomplete floor plan scan within a system involves, for each line segment within a known line segment, extending the line segment by a known amount to create an extended line segment associated with line segments having the same orientation, creating a second line segment of a second size that is perpendicular to and contacts the line segment, adding the extended line segment and the second line segment to a set of extended line segments, for each pair of extended line segments within the set of extended line segments, adding possible intersections where the line segment orientations associated with the extended line segments are similar to a set of possible intersections, finding possible edge paths between two line segments associated with an extended line segment, and creating a cost graph including a vertex set and an edge set, where the vertex set includes vertices representing the midpoints of each line segment, and the edge set includes edges, where each edge includes an intersection within the set of possible intersections, a start vertex of the intersection, an end vertex of the intersection, and a cost, and the cost includes the length from the start vertex to the end vertex and an aesthetic cost, creating a cost graph, and iteratively, until the floor plan graph forms a loop, selecting the edge with the lowest cost in the cost graph, providing the selected edge, adding the selected edge to the floor plan graph, and deleting each edge having a start vertex equal to the selected edge start vertex.

[0013] Receiving feedback from a user indicating the accuracy of a floor plan using a LiDAR-enabled user device within a system includes showing a part of a structure corresponding to an incomplete floor plan scan on a display and receiving intentional marks placed on the display.

[0014] The processor can process the intentional marks to create directed line segments and add the directed line segments to known line segments. The processor adjusting an optimized floor plan based on a display can include regenerating the optimized floor plan.

[0015] The embodiments also include a non-transitory machine-readable storage medium configured with data and instructions that, when executed by at least one processor, cause one or more devices to perform a method of extracting a floor plan from an incomplete floor plan scan consisting of known line segments. The method includes using a processor to generate an optimized floor plan by applying heuristics to the incomplete floor plan scan, displaying the optimized floor plan by a LiDAR-enabled user device, receiving feedback indicating the accuracy of the floor plan using the LiDAR-enabled user device, and using a processor to adjust the optimized floor plan based on the feedback in response to the feedback.

[0016] Embodiments of the non-transitory storage medium include, for each line segment within a known line segment, extending the line segment by a known amount to create an extended line segment associated with line segments having the same orientation, creating a second line segment of a second size that is perpendicular to and touches the line segment, adding the extended line segment and the second line segment to a set of extended line segments, for each pair of extended line segments within the set of extended line segments, adding possible intersections where the line segment orientations associated with the extended line segments are similar to a set of possible intersections, finding possible edge paths between two line segments associated with an extended line segment, and creating a cost graph including a vertex set and an edge set, where the vertex set includes vertices representing the midpoints of each line segment, and the edge set includes edges where each edge includes an intersection within the set of possible intersections, a start vertex of the intersection, an end vertex of the intersection, and a cost, where the cost includes the length from the start vertex to the end vertex and an aesthetic cost, and can also include repeatedly selecting, in the cost graph, an edge having the lowest cost until a floor plan graph forms a loop, providing the selected edge, adding the selected edge to the floor plan graph, and deleting each edge having a start vertex equal to the selected edge start vertex.

[0017] Embodiments of the non-transitory memory medium include that the LiDAR-enabled user device further includes a display, and that the display receives feedback from a user using the LiDAR-enabled user device indicating the accuracy of the floor plan, shows a part of the structure corresponding to an incomplete floor plan scan, and receives intentional marks placed on the display.

[0018] Embodiments of the non-transitory memory medium also include that a floor plan optimized for correcting the location of facilities at locations represented by incomplete floor plan scans is used. The above and other aspects of the present invention will be better recognized and understood when considered in conjunction with the following description and the accompanying drawings. The following description shows various embodiments and many of their specific details, but is presented by way of example and not limitation. Many substitutions, modifications, additions, or arrangements may be made within the scope of the embodiments, and the embodiments include all such substitutions, modifications, additions, or arrangements.

[0019] The non-limiting and non-exhaustive embodiments of the present embodiment will be described with reference to the following drawings.

Brief Description of the Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4A

Figure 4B

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Embodiments for Carrying Out the Invention

[0021] Hereinafter, representative embodiments of a method, a non - transitory machine - readable medium, and a system that are particularly applicable to a system and method for creating a better floor plan from an incomplete floor plan scan are disclosed. The described embodiments implement one or more of the described technologies.

[0022] In the following description, many specific details are set forth in order to provide a thorough understanding of the present embodiment. However, it will be apparent to those skilled in the art that these specific details are not necessarily required to practice the present embodiment. Also, well-known materials or methods are not described in detail to avoid obscuring the present embodiment. The phrase "one embodiment", "an embodiment", "one example" or "an example" means that a particular feature, structure or characteristic described in connection with that embodiment or example is included in at least one embodiment of the present embodiment. Accordingly, the appearances of the phrases "in one embodiment", "in an embodiment", "one example" or "an example" in various places throughout this specification are not necessarily all referring to the same embodiment or example.

[0023] Without departing from the scope of the present disclosure, modifications, additions or omissions may be made to the systems, apparatuses and methods described herein. For example, components of the systems and apparatuses may be integrated or separated. Also, the operations of the systems and apparatuses disclosed herein may be performed by more or fewer components or other components, and the methods described may include more or fewer steps or other steps.

[0024] Furthermore, the steps may be performed in any suitable order. For convenience, the present disclosure may be described using relative terms, including, for example, left, right, topmost, bottommost, front, rear, upper, lower, above and below, and others. These terms are used for illustrative purposes only and do not imply any limitation in any way.

[0025] Furthermore, it should be understood that the figures provided with this specification are for the purpose of explanation to those skilled in the art and that the drawings are not necessarily drawn to scale. Applicants wish to note for any reader of the patent and any patent issued with respect to the present application that unless the terms "means for" or "step for" are expressly used in a particular claim, none of the appended claims or claim elements are intended to invoke 35 U.S.C. § 112, ¶ 6.

[0026] Embodiments according to the present disclosure may be implemented as an apparatus, a method, or a computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects that may sometimes be referred to as a "system." The present disclosure may also take the form of a computer program product embodied in any tangible medium having computer-usable program code embodied in the medium.

[0027] Any combination of one or more machine-usable or machine-readable media may be used. By way of example, the machine-readable media may include one or more of a portable computer diskette, a hard disk, a random access memory (RAM) device, a read-only memory (ROM) device, an erasable programmable read-only memory (EPROM or Flash memory) device, a portable compact disc read-only memory (CDROM), an optical storage device, and a magnetic storage device. The computer program code for carrying out operations of the present disclosure may be written in any combination of one or more programming languages.

[0028] The flowcharts and block diagrams in the flow diagrams illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of code that includes one or more executable instructions for implementing the specified logical function. It should also be noted that each block and combination of blocks in the block diagrams and / or flowchart diagrams can be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions. These computer program instructions can be stored in a machine-readable medium that, when the instructions stored in the machine-readable medium cause a computer or other programmable data processing apparatus to function in a specific manner, results in a manufacture that includes instruction means for implementing the functions / operations specified in one or more blocks of the flowchart and / or block diagram.

[0029] As used herein, the terms "comprises," "comprising," "including," "include," "has," "having," or any other variation thereof are intended to cover a non-exclusive inclusion. For example, a process, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements, but may include other elements not expressly listed or inherent to such process, article, or apparatus.

[0030] Also, unless expressly stated to the contrary, "or" refers to an inclusive or and not an exclusive or. For example, the condition "A or B" is satisfied by any one of the following: namely, A is true (or exists) and B is false (or does not exist), A is false (or does not exist) and B is true (or exists), and both A and B are true (or exist). As used herein, "program" is used broadly to include applications, kernels, drivers, interrupt handlers, firmware, state machines, libraries, and other code written by a programmer (also called a developer) and / or automatically generated. "To optimize" means to improve and does not necessarily mean perfect. For example, it may be possible to further improve an optimized program or algorithm.

[0031] Furthermore, examples or illustrations presented herein should in no way be construed as limiting, restricting, or explicitly defining the definition of any one or more terms used therewith. Rather, these examples or illustrations should be considered as being presented with respect to one particular embodiment and as being merely illustrative. One of ordinary skill in the art will recognize that any one or more terms used therewith, whether or not shown with these examples or illustrations, may encompass other embodiments, and that all such embodiments are intended to be included within the scope of those one or more terms. Expressions that indicate such non-limiting examples and illustrations include, but are not limited to, "for example," "as an example," "e.g.," and "in one embodiment."

[0032] The technical features described in this specification will be apparent to those skilled in the art and will be apparent in some respects to a wide range of careful readers. Some embodiments address technical activities rooted in computing technologies, such as the creation of floor plans from 3D point clouds, using computing technologies that are far more efficient, using far less processing power and far less computer time. Other advantages based on the technical features of this disclosure will also become apparent to those skilled in the art from the description provided. Overview By simply walking through a building and surveying the area with a camera / depth scanner device, it is possible to create a rough floor plan using floor plan detection such as Augmented Reality / LiDAR (Augmented Reality / Light Detection and Ranging) that enables real-time detailed 3D point cloud capture with reasonable accuracy and precision. This 3D point cloud can be automatically structured to create a 3D mesh. As described in Patent Application No. 17 / 459,084, filed on September 15, 2021, and incorporated herein by reference in its entirety for all purposes, a set of 2D line segments can be extracted that indicate the general outline of important structures and mesh shapes. Referring to 100 in FIG. 1, even the best LiDAR representation often contains minor errors. According to various embodiments, a depth measurement system such as LiDAR is used to capture the 3D point cloud or 3D mesh of the environment simultaneously with the same environment being captured in a 2D Augmented Reality (AR) environment. Various embodiments then use a combination of user input and machine learning to place annotations on the AR environment. These annotations are placed using 2D coordinates from the 3D LiDAR system. Looking at FIG. 1, it may appear that the closest adjacent lines can be easily intersected, but this problem can be computationally much more difficult. The goal is to draw a set of paths that represent the outline of each room. To do this, it is necessary to determine a path through a graph that reaches each vertex exactly once. However, this is called the Hamiltonian path problem and is NP-complete. A closely related problem that can be used to determine the floor plan is the "longest path problem in a graph". This approach can avoid shortcuts and is theoretically faster, but it is still NP-hard. Therefore, neither method will create a connected floor plan within any reasonable amount of usable time.

[0033] Since the theoretical results are not currently computationally feasible, the various embodiments disclosed herein describe systems and methods for improving incomplete floor plan scans. Floor plans are constructed wall by wall using a user-held LiDAR device having a display screen. The user scans the room and places marks on the display screen indicating the locations of the walls. When new walls are determined by user marks, LiDAR calculations, and heuristics, the user can view the current floor plan. If the floor plan indicates an error, the user can place another mark to more precisely specify the location of the wall. This enables the creation of floor plans that previously required computer processing power and time that were too computationally expensive to perform effectively. There is another solution to this problem where the user drops pins at each corner of the room, but this pin annotation must be placed in order while the person walks around the room. It is something like trying to wrap a room with string or ribbon, and if a pin is missing, the process must be restarted from the beginning.

[0034] A distinct advantage of the applicant's approach is that the user can place the markings in any order. If the user is given feedback indicating that a previous placement was insufficient, the user simply adds a marking. This is far more interactive, while "pin dropping" requires significant forethought.

[0035] Figure 3 shows a generalized example of a suitable machine environment 300 in which the described embodiments can be implemented. Since the present disclosure is implementable in a variety of general-purpose or special-purpose machine environments, machine environment 300 is not intended to suggest any limitation as to the scope of use or functionality of the present disclosure.

[0036] Referring to FIG. 3, core processing is shown by the core processing 330 box. The machine environment 300 includes at least one central processing unit 310, a GPU 315, and a memory 320. The central processing unit 310 can execute machine-executable instructions and can be a physical processor or a virtual processor. The memory 320 can be volatile memory (e.g., registers, caches, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of the two. The memory 320 stores software 385 that implements the described method of extracting a floor plan from an incomplete floor plan scan.

[0037] The machine environment may have additional features. For example, the machine environment 300 includes a storage 340 and a communication connection 370, and the communication connection 370 can include one or more input / output devices 350, one or more network connections (such as wired, wireless, etc.) 360, and other communication connections (not shown). The communication connection 370 may include one or more wireless transceivers for performing wireless communication and / or one or more communication ports for performing wired communication. In an embodiment, the communication device 370 can be configured to receive building features captured using 2D sensors and 3D sensors. The input / output device may include a two-dimensional camera 352. It may also include a depth sensor 354 such as a LiDAR (Light Detection and Ranging) system, and a touch screen 356 that allows a user to input information and display image information. An interconnection mechanism (not shown) such as a bus, a controller, or a network interconnects the components of the machine environment 300. Typically, an operating system software (not shown) provides an operating environment for other software executed within the machine environment 300 and coordinates the activities of the components of the machine environment 300. The computing system within the machine environment may be distributed, and each part of the software 385 may be executed on a different CPU. For example, in some embodiments, each part of the software 385 may be executed on a device in the same location as the depth sensor 354 or the touch screen 356, while other parts of the software may be executed on a separate server or virtual machine (not shown) accessible via the network connection 360.

[0038] The storage 340 may be removable or non-removable and can be a magnetic disk, magnetic tape or cassette, CD-ROM, CD-RW, DVD, flash drive, or any other medium that can be used to store information and is accessible within the machine environment 300. The storage 340 stores instructions of software such as the region of interest detection software 385 that implements a method for extracting a floor plan from an incomplete floor plan.

[0039] The input device 350 can be a device that enables a user or other device to interact with the machine environment 300, such as a touch input device like a keyboard, a camera 352, a depth measurement system 354 such as a LIDAR system, a microphone, a mouse, a pen, or a trackball, a scanning device, a touch screen 356, or other devices that provide input to the machine environment 300. For audio, the input device 350 may be a sound card or similar device that accepts audio input in analog or digital form, or a CD-ROM reader that supplies audio samples to the machine environment. The output device 350 may be a touch screen, a display, a printer, a speaker, a CD writer, or other devices that provide output from the machine environment 300. A compass / magnometer 390 may be included so that the device can determine in which direction an image is captured by the camera 352 and the depth sensor 354.

[0040] The communication connection 370 enables communication via a communication medium with another computing / machine entity. The communication medium transmits machine-executable instructions, compressed graphics information, or other data as a modulated data signal. The communication connection 370 can include an input / output device 350 and an input / output device that enables the client device to communicate with another device via the network 360. The communication device may include one or more wireless transceivers for wireless communication and / or one or more communication ports for wired communication. These connections can include network connections that can be wired or wireless networks such as the Internet, an intranet, a LAN, a WAN, a cellular network, or other types of networks. It should be understood that the network 360 may be a combination of multiple different types of wired or wireless networks. The network 360 may be a distributed network having multiple computers that may be building controllers that function in cooperation. The computing connection 370 may be a portable communication device such as a wireless handheld device, a cellular phone device, etc.

[0041] A machine-readable medium is any available non-transitory tangible medium accessible within a machine environment. By way of example, and not limitation, in the case of machine environment 300, the machine-readable medium can be memory 320, storage 340, a communication medium, and any combination of the foregoing. Machine-readable storage medium 365 can be used to store instructions 375 and data 380. A data source can be a computing device such as a general-purpose hardware platform server configured to send and receive information via communication connection 370. Machine environment 300 can be an electric controller directly connected to various resources such as HVAC resources and having CPU 310, GPU 315, memory 320, input device 350, communication connection 370, and / or other features shown in machine environment 300. Machine environment 300 can be a series of distributed computers. These distributed computers can include a series of connected electric controllers.

[0042] Although some of the operations of the methods of the present disclosure are described in a particular order for convenient presentation, it should be understood that this presentation method includes permutations unless a particular ordering is required by the specific expressions described below. For example, the operations described sequentially can be permuted or performed in parallel. Also, for simplicity, the accompanying drawings may not show the various ways in which the methods, apparatuses, and systems of the present disclosure can be used with other methods, apparatuses, and systems. Further, in the description, terms such as "determine," "construct," and "specify" may be used to describe the techniques of the present disclosure. These terms are highly abstracted of the actual operations performed. The actual operations corresponding to these terms will vary depending on the particular embodiment and will be readily identifiable by those skilled in the art.

[0043] In addition, data generated from any of the methods of the present disclosure can be created, updated, or stored on a tangible machine-readable medium (e.g., one or more CDs, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as a hard drive)) using a variety of different data structures or formats. Such data can be created or updated on a local computer or via a network (e.g., by a server computer), or stored and accessible in a cloud computing environment.

[0044] Examples of the back and front of an exemplary user device having a depth scanner and a camera are shown in 400A of FIG. 4A and 400B of FIG. 4B. The systems and methods described herein can be implemented using a mobile phone 400A, another portable computing device, or a computing device having a 2D camera 405A and a depth scanner 410A. In some embodiments, the system includes a user input device. The user input / output device may be a touch screen 405B. The user input device may be associated with a device different from the device housing the depth scanner and the camera.

[0045] To capture a space, in some embodiments, a user walks around an area in a space while pointing a device having both a camera and a depth scanner (e.g., FIGS. 4A, 4B) from multiple viewpoints. A 3D point cloud scan can be created using a LiDAR depth scanner. Augmented Reality / Light Detection And Ranging (AR / LiDAR) enables a person to walk through a building and simply survey the area with a 2D / 3D scanner, capturing a detailed 3D point cloud in real-time with reasonable accuracy and precision. This 3D point cloud can be automatically structured to create a 3D mesh. However, a typical LiDAR mesh appears as an unstructured “triangle soup” with a lot of noise and artifacts, making it extremely difficult to distinguish important aspects of a scene, such as walls, from furniture, appliances, and other substances present. Also, the amount of raw detail makes it difficult to extract regions of interest, such as walls, windows, and doors. For example, a building contains many flat “wall-like” surfaces that are not walls, such as cabinets, bookshelves, and desks, making it difficult to determine geometric criteria for specifying walls from non-walls. The definition of these features depends heavily on context.

[0046] In 500 of FIG. 5, an example of a 2D line segment 505 having an orientation (direction) 510 used to create an optimized floor plan is shown. Here, the orientation is at one end, but the orientation can be at any position along the line segment. One way to create an incomplete floor plan scan is to capture the 3D environment while intelligently detecting features that are not part of the floor plan. A depth measurement system such as LiDAR can be used to capture the 3D point cloud or 3D mesh of the environment simultaneously with the same environment in a 2D augmented reality (AR) environment. The 2D capture from the camera presents a representation of the 3D depth geometry using a 2D transformation of the 3D grid. The transformation is provided by software associated with the camera, depth scanner, or both. This scan includes relevant readings from the sensor, including the orientation and compass bearing. The user can place 2D display annotations in the area indicating the wall on the display. Next, a pre-trained machine learning algorithm can be used to place the annotations added by the user within the 3D mesh environment. The annotations, in this case, serve as a hint to identify the approximate location of features that the user is interested in, such as a wall. The placement of the annotations does not have to indicate exactly where the feature is, but rather suggests an approximate area within the 3D mesh to search for the feature of interest. After the 3D mesh is completed, in both the real-time preview and post-processing steps, the 2D annotations are converted to specific locations in 3D space, and the 2D annotations and orientations are placed. The conversion can include, in part, averaging the locations in 3D space, averaging the locations in 2D space, or a combination of the two. In some cases, the placement is delayed until post-processing, thereby enabling a significant improvement in accuracy and consistency. The area around the 3D scan annotation, which can be a mesh surface within the 3D mesh, is examined to see if there is a mesh surface with a similar orientation to the mesh surface in the 3D scan annotation. These mesh surfaces may be adjacent mesh surfaces. If an adjacent mesh surface with a similar orientation is found, the adjacent surface of the found mesh surface is examined, and so on.These mesh surfaces having similar orientations are then marked as part of the region of interest. After a part of the region of interest is found, it can be determined that this part is the floor plan part. This floor plan part can be flattened to the 2D marker 500 by flattening the marker so that the flattened part is parallel to the floor. An incomplete floor plan such as that shown in FIG. 1 can be calculated based on the compass orientation and other orientations automatically determined by the 2D camera and LiDAR-generated 3D scan.

[0047] A system for creating an incomplete floor plan and on which annotations are placed is disclosed in U.S. Patent Application No. 17 / 459,048, filed September 15, 2021, which is hereby incorporated by reference in its entirety for all purposes. FIG. 6 shows an example of a graphical representation of a set of known two-dimensional line segments 605 having positions and orientations 610 that may represent an incomplete floor plan as used herein. This incomplete floor plan includes gaps 615 that need to be filled to create a more useful floor plan. Embodiments Referring to FIG. 7, a flowchart 700 depicts a method that can be used by a device in the embodiments disclosed herein. The operations of method 700 shown below are intended to be exemplary. Depending on the embodiment, method 700 may be performed with one or more additional operations not described and / or without one or more of the operations described. Also, the order of the operations of method 700 shown in FIG. 7 and described below is not intended to be limiting.

[0048] Depending on the embodiment, method 700 may be implemented by a processor and a program 385 stored in a memory element 330 within one or more processing devices (e.g., digital processors, analog processors, microprocessing units, digital circuits designed to process information, and / or mechanisms for electronically processing information), such as within a core processing element that includes a central processing unit 310 and a memory element 320. The one or more processing devices may include one or more devices that execute some or all of the operations of the method in response to instructions electronically stored on an electronic storage medium. The one or more processing devices may include one or more devices configured by hardware, firmware, and / or software specifically designed for performing one or more of the operations of the methods and systems described herein. Operation 705 discloses using a processor to generate an optimized floor plan using the methods and systems disclosed herein. Operation 710 discloses displaying the optimized floor plan. This display may be on a LiDAR-enabled device, such as 400B, associated with a computing environment, which may or may not be a touch screen device or a different device.

[0049] Figure 8 discloses a display seen on the display screen of a depth scanner / camera device or a different type of computer having a display screen. This display may be a background of a part of a structure being scanned, such as a part of a room shown at 800 in Figure 8, something visible through a viewfinder. The display can also show a mark 805 made by the user on the display screen that marks the walls previously used to create the optimized floor plan 810. For example, the user can drag a finger or a stylus across the screen, leaving a colored mark 805 or other indicia indicating the shape traced thereby. Depending on the implementation, the mark 805 may be associated with the location of the surface on which the mark 805 is drawn in three-dimensional space. Subsequently, when the location of the camera device (e.g., camera 352) changes and the view of the marked surface on the display changes, the image of the mark 805 can be transformed in a similar manner, such as by translation, rotation, scale, skew, etc., so that the marked and displayed state remains the same as when the surface is visible on the display.

[0050] The optimized floor plan, as shown in the exemplary preview panel 810, may not be the best possible optimization. Here, the floor plan being created appears to be like two triangular rooms, and at this point, it looks like an error because three walls of a single room have been scanned (the first two walls and their intended markings are not shown). Ideally, therefore, the floor plan should show a single triangle since the fourth wall has not yet been scanned. The two triangles shown indicate that additional intentional markings are needed to provide more information for more accurately determining the floor plan. In an embodiment, the display can also show the direction in which a particular video was taken, or the direction in which the camera device is pointed relative to the floor plan being created when the view on the display 800 is a live virtual reality view of the space where the view is mapped. This is shown by the arc 815, with the vertex indicating the user's position and the arc indicating the camera's field of view. This enables the user to position themselves within the space to determine how accurate the more optimized floor plan display 810 is.

[0051] In operation 715, the processor receives feedback regarding a more optimized floor plan. A user looking at a more optimized floor plan, such as 810, can notice that the floor plan is incorrect and thus can mark another intentional mark, for example, to give more feedback to a floor plan extraction program. In 900 of FIG. 9, an example is shown of another view of the same wall shown in FIG. 8, but with the depth scanner moved to show additional portions that were not previously within the view. Next, the user can place another intentional mark 905 on the display. The device (or a processor connected to the device via a wired or wireless connection) can then process the received information to determine new floor plan line segments that can be used to more accurately determine the floor plan. In some embodiments, a plurality of depth measurements captured by a depth sensor may be obtained to change the intentional mark into a line segment. These depth measurements can be obtained from a program associated with depth measurement capture. The depth measurements define the distance from the depth sensor to each point in space using a three-dimensional grid.

[0052] An intentional mark (initially created two-dimensionally) can be mapped onto a three-dimensional grid using depth measurements to form three-dimensional rays. The two dimensions are detectable in a two-dimensional grid (associated with the display), which can then be converted to three-dimensional mesh grid locations using available software, such as with a 3D mesh system, a 2D camera system, some combination, etc. For points detected within the intentional mark, for at least some of the points, the two-dimensional points are then converted to rays having a three-dimensional tail that intersects the three-dimensional shape to which the region of interest pertains. Next, the rays are traced to determine where in the 3D world they hit the mesh, creating 3D rays where the points of the rays intersect the wall. Next, a plurality of 3D rays along the intentional mark can be averaged at points along each ray to determine a flattened 2D directed line segment. In operation 720, if this 2D line segment, for example the feedback, is arranged such that the processor can understand it, a more optimized floor plan is adjusted using the feedback. This optimization 910 is then shown on the display 900. In some cases, the optimization may be shown within a limited area 915 within the display. In this case, the limited area 915 is outlined by a dotted line. Other limited areas and contour displays are also included within the scope of the present disclosure. If, when shown, the more optimized feedback 910 is still incorrect, the user can place another intentional mark until the floor plan looks correct.

[0053] FIG. 10A at 1000A and FIG. 10B at 1000B receive feedback and adjust a more optimized floor plan based on the feedback, and an example flowchart for explaining this in more detail is shown. In operation 1005, wall markings are received. The reception of wall markings is described with reference to FIGS. 8 and 9 and the surrounding text. In operation 1010, line segments and directions are determined. The determination of new line segments can include averaging multiple positions and directions 1015. These line segments represent the wall portions in the incomplete part of the floor plan. The direction of the line is associated with the line segment and points towards the estimated center of the room. These are described in more detail with reference to FIGS. 5 and 6 and the surrounding text. In operation 1020, this new line segment is added to the known line segments. FIG. 11 at 1100 shows an example of known line segments in graphical form that can be connected to form a floor plan. At 1110, a single line segment with a direction is shown. This line segment may be added because it may generate a non-optimal floor plan together with other line segments such as the line segment shown at 810 by line segment 1105. In operation 1025, each line segment can be extended by a certain amount, thereby creating extended line segments. For example, the line segment may be extended by a fixed amount, an amount within a range, different line segments may be extended by different amounts, or another method may be used. In some cases, a threshold amount that defines the maximum amount by which a line can be extended may be set. FIG. 12 at 1200 shows an example of the line segment of FIG. 11 that has been extended by a certain amount. In operation 1020, these extended line segments are added to the set of extended line segments.

[0054] This graph of the extension segments represents possible paths, but does not represent all possible paths connecting the segments. In some cases, the segments may be placed in the wrong location. For example, the original graph may have multiple segments that should be connected to form a single segment, but are incorrectly offset. An example of such segments is shown in FIG. 11 with reference to segments 1105 and 1110. These two offset segments represent the same wall. When these two lines are extended to 1205, 1210, these two lines become parallel. Other offset segments may capture actual surfaces including columns, narrow corridors, etc. It is important to be able to distinguish between errors and actual architectural features. For this purpose, in operation 1027, lines perpendicular to the segments are created. These perpendicular lines may be created for each segment, each extended segment, or as a subset of the segments. The perpendicular lines may be created at the ends, in the middle, or at some other location of the segment. An example of such a perpendicular line intersecting another segment is shown at 1300 in FIG. 13. Segment 1310 (e.g., segment 1110, extended segment 1210) has a perpendicular line 1315 created at its right end 1320. The length of this perpendicular line 1315 may be a fixed length, may depend on the length of segment 1310, or may be a range of lengths, etc.

[0055] In operation 1030, next, the intersections (e.g., 1320, 1325) of these segments and the extended segments 1305, 1310 are found. For this purpose, a sweep line algorithm such as the Shamos-Hoey algorithm, the Bentley-Ottman algorithm, or a different algorithm can be used. Intersections that are impossible wall intersections due to differences in orientation are removed. FIG. 14 shows an example of a graph 1400 that is a possible set of intersections passing through the segments of FIG. 11. This figure shows the original segments such as 1405 and the extended segments such as 1410, along with a graph of the extended segments such as 1410, 1415 meeting at an intersection such as 1420.

[0056] Next, examine graph 1400 to find a good path through it. To construct a floor plan from line segments that match the actual floor plan, it is then necessary to select one exit intersection and one entry intersection for each line segment. The problem is how to select which line segments will generate the floor plan. For this purpose, this method finds a path (original line segments, e.g., 1405) through a graph that reaches each vertex exactly once. In other words, this approach aims to include each actual line segment without shortcuts (e.g., the line segments seen in FIG. 11). This can provide a loop that passes through all actual line segments. However, finding such a loop is a very difficult problem to solve, known as the well-known Hamiltonian cycle. Determining the existence of a closed path through line segments is NP-complete even for graphs with limited degrees. Another heuristic that can be used is the "longest path in a graph," which is also NP-complete. These two theoretical solutions have the difficult problem of hard exponential time complexity, where the time required for solution doubles with each additional element, as shown by the graph of 1500 in FIG. 15.

[0057] The various embodiments disclosed herein use heuristics that yield solutions in polynomial time. This is done by using the concept of "closeness" between line segments. This makes sense from a user experience perspective as the user expects the markers to be connected and thus the markers are placed in close proximity to each other. Both the distance between line segments and the angle between the selected close line segments are indicators that the two line segments should be connected to each other. In operation 1035, a cost graph is created. This cost graph can include a vertex set 1040 and an edge set 1045. The vertex set can include some or all of the existing line segments (such as the original line segments shown in FIG. 11). These existing line segments can be represented by two-dimensional points, i.e., vertices. This point can be the midpoint of the line segment or a different position within the existing line segment. The edge set can include possible intersection / connection relationships (such as 1420) between two of the line segments represented in the graph using extension line segments and lines perpendicular to the line segments to find intersections. The edge set value can include the actual intersection location between two extension line segments and the two vertices of the non-extension line segments whose extensions formed the intersection. Depending on the embodiment, one may also store non-extension line segments as part of the edge set and so on.

[0058] In operation 1050, the cost of an edge is determined. The cost of the edge can be the length from the first vertex at the edge (e.g., 1405) to the second vertex (e.g., 1425) passing through the intersection (e.g., 1420). The cost can be a value obtained by multiplying the vertical distance by a constant, a value obtained by multiplying the horizontal distance by a constant, etc. The cost can be any combination of the above. In operation 1060, the edge with the lowest cost (e.g., the shortest intersection line segment) is determined. In operation 1065, two line segments (e.g., the original line segments 1115, 1120) having the shortest intersection are connected. In 1600 of FIG. 16, an example of the edge with the lowest cost in the connected FIG. 14 is shown. In an example of the connection, a line segment is created that continues from a point on one of the line segments of the original line segments represented by the selected edge, passes through the intersection 1605 (represented by the selected edge), and reaches a point on the other line segment 1609 (represented by the selected edge). The connection of the line segments is described in more detail in FIG. 19 and the surrounding text. In operation 1070, the connected line segment is added to a more optimized floor plan. In operation 1075, from the graph, the edge between the two vertices to be connected and the appropriate ends are detected together with each edge (line segment) attached to the connected ends of the vertices. Examples of the edges to be deleted when a given line segment is connected are 1610, 1615, and 1620. In 1700 of FIG. 17, a vertex-edge diagram is shown after the second line segment-intersection set 1705 is selected and the cost graph line segment having the same start position as the edge with the lowest cost is deleted.

[0059] At decision point 1080, it is determined whether a loop has been created. If a loop has been created, in operation 1090, a floor plan is displayed. If a loop has not been created, at step 1085, if there are no vertices left in the cost graph, at decision point 1087, the vertex that has reached a dead end is deleted. This will be described in more detail with reference to FIGS. 25-29 and the surrounding text. Next, in operation 1060, the construction of the floor plan starts again. In some embodiments, after displaying the floor plan in step 1090, the method may receive additional information from the user, such as an additional different intentional mark 905 for example. In some embodiments, the method may receive changed scan data (such as a rescan after a new change has been made to the environment by moving furniture or closing a door) from the camera 352 or the depth sensor 354. In some such embodiments, methods 1000A, 1000B can recalculate the floor plan, for example, by looping back to step 1005.

[0060] FIG. 18 shows at 1800 the cost graph shown in FIG. 14 with the outline of a more optimized floor plan 1805 drawn thereon. FIG. 9 shows at 910 the display of the floor plan after further optimization after three walls have been scanned. In operation 1095, a decision is made to change the nature of the space represented by the floor plan constructed herein and / or to change the space in which the floor plan is located using the floor plan constructed herein. Many changes are possible. Among them, some possible changes are to determine where sensors can be placed, then place the sensors in the building, use the floor plan to determine the floor area, use the floor area to determine how much heating and cooling is required, use the required amount of heating and cooling to determine the size of the heating and cooling system that will be placed in the building, purchase the heating and cooling system, and then install the heating and cooling system in the building.

[0061] An example of a flowchart is shown in 1900 of FIG. 19 for more detailed explanation of performing a connection (for example, operation 1065) on one end of two connected line segments. In some cases, two lines that are offset from each other by a certain amount are connected. In such cases, it is possible to determine whether to connect those lines by assuming that the two lines should be joined as a single line, or to connect them by forming a zigzag corner. In operation 1905, a first angle between the closest ends of the two line segments to be connected is determined. An example of measuring the first angle 2015 between the closest ends 2010, 2020 of two line segments (2005-2010 and 2020-2025) to be connected is shown in 2000 of FIG. 20. In operation 1910, a second angle between the farthest ends of these line segments is determined. An example of measuring the second angle 2115 between the farthest ends 2105, 2125 of two line segments (2105-2110 and 2120-2125) to be connected is shown in 2100. In operation 1915, it is determined whether the line segments should be connected by making them a single straight line. The line segments need to be connected in this way when the first angle is less than (or equal to) a first threshold value and the second angle is less than (or equal to) a second threshold value. The first angle threshold value and the second angle threshold value can have a wide range of values. An example of connecting by connecting the farthest point 2205 of the first line segment to the farthest point 2225 of the second line segment by a single straight line 2230 to make two points a single straight line is shown in 2200 of FIG. 22.

[0062] In operation 1920, the line segment is zigzagged. The "zigzagging" relates to the human concept regarding the building method of buildings where buildings are usually constructed at right angles. The difference is very likely to be small connecting parts or columns that are not fully represented by the line segment. With that in mind, depending on the implementation form, the zigzag is selected when the first angle (e.g., 2015) exceeds the threshold or the second angle (e.g., 2115) exceeds the threshold. As shown in the example shown at 2300 in FIG. 23, one of the angles of the line segment 2340 exceeds the threshold. To zigzag the line, the x - y axis 2335 is created using the intersection of the two closest endpoints 2310 and 2320. This x - y axis can intersect at a 90 - degree angle or other selected angles. As shown in the example shown at 2400 in FIG. 24, at the selected fold axis, the line from the intersection of the x - y axis 2435 to one of the adjacent endpoints 2420 is divided in half (or divided at some other ratio), and then, by the selected fold axis line 2440 at one end (at a certain ratio) and the non - selected axis line 2445 parallel to the non - selected axis line drawn at the other end (100 - ratio), a zigzag is drawn between the two close points 2410, 2420. Other methods can also be used to draw a line between these two discontinuous points (e.g., 2410, 2420).

[0063] In some cases, the floor plan scan cannot form a loop. FIG. 25 shows an example of a floor plan 2515 having the original line segment 2505 and the extension line segment 2510 where the formation of the loop has failed. FIGS. 26 and 27 show two close - ups of the unconnected line segments 2520. As shown in the close - up of the intersection 2700, the two markers 2605 and 2610 are not connected because the marker 2705 is slightly on the inappropriate side of the intersection 2710.

[0064] To explain the concept of what can be connected and what cannot be connected, reference numeral 2800 in FIG. 28 shows an example of two line segments that have the possibility of being connected. Reference numeral 2900 in FIG. 29 shows an example of two line segments that cannot be connected. In these two figures, the arrows on the line segments are to the left with respect to the direction. That is, these indicate what is inside the "room". If the line segments can intersect, the intersection point 2815 may be in front of one line segment 2810 that may intersect and behind the other line segment 2805 that may intersect. The line segments may not intersect if both line segments are in front of or behind the intersection point. Line segments 2905 and 2910 are both behind the intersection point 2915. In such a case, the directions of the two marks are different, and thus the line segments cannot be connected. This can be visualized as an incorrect line segment 2905 outside the room defined by line segments 2920 and 2910 here. These line segments that can be connected only once lead to a dead end. As can be seen with reference to FIG. 27, an incorrect direction occurs only slightly, deviating slightly from the range where one line segment is properly connected. These minor misaligned line segments occur due to minor errors that can accumulate. If this line segment that can be connected only on one side is deleted, the proper line segments that define the floor plan can be connected up to, for example, 2920. To correct this problem, referring to FIGS. 10A and 10B, a floor plan loop is not created in operation 1080, and no vertices to be connected are left at 1085. This indicates that a problem has occurred. At 1087, the vertices that lead to a dead end are deleted. The cost graph at this point (e.g., 1035, 1075) mirrors the floor plan graph with missing vertices. The cost graph is scanned to find vertices that are connected to another vertex at only a single end. This finds vertices with a single connection as shown with reference to 2705 in FIG. 27. Depending on the embodiment, the positions of multiple vertices with a single connection can be arranged. These vertices are then deleted. After deletion, depending on the embodiment, methods 1000A, 1000B can recalculate the floor plan by looping back to, for example, step 1005, step 1060, or different steps.

[0065] While various exemplary embodiments have been described in detail with specific reference to particular exemplary aspects, it should be understood that the present invention is capable of other embodiments and that its details are capable of modification in various obvious respects. As will be readily apparent to those skilled in the art, various changes and modifications can be made while remaining within the spirit and scope of the present invention. Accordingly, the above disclosure, description, and drawings are for illustrative purposes only and do not limit the present invention in any way, which is defined only by the claims.

Claims

1. A machine-implemented method for extracting a floor plan from an incomplete floor plan scan consisting of known line segments, comprising: using a processor to generate a more optimized floor plan; displaying the more optimized floor plan by means of a user device having a 3D depth scanner and a 2D camera; receiving feedback indicating the accuracy of the more optimized floor plan by using the 3D depth scanner and the 2D camera; using the processor to adjust the more optimized floor plan based on the feedback in response to the feedback.

2. Receiving feedback indicating the accuracy of the floor plan includes receiving wall markings from readings from the 3D depth scanner and the 2D camera, the wall markings including a plurality of positions and a plurality of orientations of locations in 2D and 3D, the wall markings being indicated on the user device by the user. The machine-implemented method according to claim 1.

3. The machine-implemented method according to claim 2, wherein the processor processes a plurality of positions and a plurality of orientations of locations in 2D and 3D to determine a directed line segment, and the directed line segment is added to the known line segment.

4. The machine-implemented method according to claim 3, further comprising using both depth scanner measurements and camera measurements to determine an average of the plurality of positions and the plurality of orientations in order to determine a directed line segment.

5. Applying a heuristic to the floor plan includes: for each line segment within the known line segments, extending the line segment by a known amount to create an extension line segment associated with the line segments having the same orientation; and creating a second line segment of a second size that is perpendicular to and in contact with the line segment; adding the extension line segment and the second line segment to a set of extension line segments; for each pair of extension line segments within the set of extension line segments, adding possible intersections where the line segment orientations are similar to a set of possible intersections; and finding possible edge paths between two line segments associated with the extension line segment. Creating a cost graph that includes a vertex set and an edge set, where the vertex set includes vertices representing the midpoints of each line segment, the edge set includes edges, each edge includes an intersection within the set of possible intersections, the start vertex of the intersection, the end vertex of the intersection, and a cost, and the cost includes the length from the start vertex to the end vertex and an aesthetic cost, including creating a cost graph. Repeatedly until the floor plan forms a loop. In the cost graph, select the edge with the lowest cost, thereby providing the selected edge, and add the selected edge to the floor plan graph. The machine-implemented method according to claim 4, wherein each edge having a start vertex equal to the selected edge start vertex is deleted.

6. The machine-implemented method according to claim 5, wherein adding the selected edge to the floor plan graph includes adding the possible edge path between the two line segments to the floor plan graph.

7. Finding a possible edge path includes Determining a first angle between the two closest ends of the two line segments. Determining a second angle between the far end of the second line segment of the two line segments and the far end of the second line segment of the two line segments. When the first angle is less than a first threshold amount and the second angle is less than a second threshold amount, connecting the first line segment of the two line segments and the second line segment of the two line segments. The machine-implemented method according to claim 6.

8. The machine-implemented method according to claim 7, wherein when the first angle exceeds the first threshold amount and the second angle exceeds the second threshold amount, zigzagging the two line segments is included.

9. The machine-implemented method according to claim 8, wherein zigzagging the two line segments includes creating a new line segment perpendicular to the first line segment of the two line segments and between the first line segments, and extending the proximal ends of the two line segments to the new line segment.

10. The machine-implemented method according to claim 9, wherein the cost further includes adding a weighting value to the cost.

11. A system for extracting a floor plan from an incomplete floor plan scan consisting of line segments, including a memory, a processor operably communicating with the memory, and a LiDAR-enabled user device having a display, wherein the processor is configured to extract a floor plan from an incomplete floor plan scan consisting of known line segments, and the extracting includes generating a more optimized floor plan by applying heuristics to the incomplete floor plan scan; displaying the more optimized floor plan by the LiDAR-enabled user device; receiving feedback indicating the accuracy of the floor plan from a user using the LiDAR-enabled user device; responding to the feedback by adjusting the more optimized floor plan based on the feedback. **Claim 12** The system of claim 11, wherein the line segments include orientations and the known line segments include line segments. **Claim 13** Applying heuristics to the incomplete floor plan includes extending each line segment within the known line segments by a known amount to create an extended line segment associated with the line segments having the same orientation; creating a second line segment of a second size that is perpendicular to and in contact with the line segment; adding the extended line segment and the second line segment to a set of extended line segments; for each pair of extended line segments within the set of extended line segments, adding possible intersections where the line segment orientations associated with the extended line segments are similar to a set of possible intersections; finding possible edge paths between two line segments associated with the extended line segment; creating a cost graph including a vertex set and an edge set, wherein the vertex set includes vertices representing the midpoints of each line segment, the edge set includes edges, and each edge includes an intersection within the set of possible intersections, a start vertex of the intersection, an end vertex of the intersection, and a cost, the cost including the length from the start vertex to the end vertex and an aesthetic cost; iteratively until the floor plan forms a loop selecting an edge having the lowest cost in the cost graph, thereby providing the selected edge; adding the selected edge to a floor plan graph; The system according to claim 12, wherein each edge having a start vertex equal to the selected edge start vertex is deleted.

14. The system according to claim 13, wherein receiving feedback indicating the accuracy of the floor plan from a user using the LiDAR-enabled user device includes the display showing a part of the structure corresponding to the incomplete floor plan scan and receiving intentional marks placed on the display.

15. The system according to claim 14, wherein the processor processes the intentional marks to create directed line segments and adds the directed line segments to the known line segments.

16. The system according to claim 15, wherein the processor adjusting the optimized floor plan based on the display includes regenerating the optimized floor plan.

17. A non-transitory machine-readable storage medium configured to include data and instructions that, when executed by at least one processor, cause one or more devices to perform a method of extracting a floor plan from an incomplete floor plan scan consisting of known line segments, the method comprising: using a processor to generate a more optimized floor plan by applying heuristics to the incomplete floor plan scan; displaying the more optimized floor plan by a LiDAR-enabled user device; receiving feedback indicating the accuracy of the floor plan using the LiDAR-enabled user device; and using the processor to adjust the more optimized floor plan based on the feedback in response to the feedback.

18. Applying heuristics to the incomplete floor plan scan includes: extending each line segment within the known line segments by a known amount to create an extension line segment associated with the line segments having the same orientation, and creating a second line segment of a second size that is perpendicular to and in contact with the line segment; adding the extension line segment and the second line segment to a set of extension line segments; For each pair of extension line segments within the set of extension line segments, adding possible intersections where the line segment orientations associated with the extension line segments are similar to the set of possible intersections, and finding possible edge paths between the two line segments associated with the extension line segment, creating a cost graph including a vertex set and an edge set, where the vertex set includes vertices representing the midpoints of each line segment, the edge set includes edges, and each edge includes an intersection within the set of possible intersections, a start vertex of the intersection, an end vertex of the intersection, and a cost, and the cost includes the length from the start vertex to the end vertex and an aesthetic cost, creating a cost graph, iteratively until the floor plan forms a loop, selecting an edge with the lowest cost in the cost graph, thereby providing the selected edge, and adding the selected edge to the floor plan graph, deleting each edge having a start vertex equal to the selected edge start vertex, the non-transitory machine-readable storage medium of claim 17.

19. The LiDAR-compatible user device further includes a display, and receiving feedback indicating the accuracy of the floor plan from a user using the LiDAR-compatible user device, including the display showing a part of the structure corresponding to the incomplete floor plan scan, and receiving intentional marks placed on the display, the non-transitory machine-readable storage medium of claim 18.

20. The more optimized floor plan is used to correct the location of equipment at the location represented by the incomplete floor plan scan, the non-transitory machine-readable storage medium of claim 19.

Images