Feature detection in AR environments

Abstract

Various embodiments described herein relate to methods, devices, and non-transitory machine-readable storage media, including one or more of the following: Locate a 2D plane correlated with a 3D mesh representing a surface of a room; Capture a virtual 2D image of the 3D mesh along a 2D segment; Find a hole in the virtual 2D image; Measure the floor of the vertical image and the width of the hole; If the hole intersects with the floor of the vertical image and is at least as wide as the door, classify the hole as a door; Otherwise, classify the hole as a window.

Description

[Technical Field]

[0001] (1) Various embodiments described herein relate to augmented reality environments, and more particularly, but not exclusively, to detecting features such as holes, doors, and windows in AR environments.

[0002] (2) Three-dimensional (3D) images cannot be directly projected onto two-dimensional (2D) images of open windows or doors because there is no 3D image to project, and the open door or open window is blank. This also applies to actual holes inside the face. Therefore, open doors and windows often cannot be detected in 3D systems. It would be useful to be able to accurately characterize features such as holes, doors, and windows in 2D and 3D systems. Summary of the Invention [Means for solving the problem]

[0003] (3) This Summary is provided to introduce in a simplified form a selection of concepts further described below in the Detailed Description section. This Summary does not identify essential features or essential features of the claimed subject matter. The innovation of the present application is defined by the claims, and to the extent this Summary is inconsistent with the claims, the claims shall control.

[0004] (4) Various embodiments described herein provide a processor-implemented method for finding features within a 3D mesh, the method including the steps of: locating a 2D segment correlated with a 3D mesh representing a surface of a room; capturing a virtual 2D image of the 3D mesh along the 2D segment; finding a hole in the virtual 2D image; finding a position of the hole in the 3D mesh; offsetting the position of the hole in the 3D mesh to form a hole offset; casting a ray from the center of the hole offset into the 3D mesh; and detecting the ray as it strikes a backside of a surface in the 3D mesh. determining whether the emitted ray hits the back side of one or less faces; classifying the hole as an error in the 3D mesh; determining a floor of the vertical image and measuring a width of the hole when the emitted ray hits the back side of more than one face; classifying the hole as a door when the hole intersects with the floor of the vertical image and is at least as wide as the door; and classifying the hole as a window when the hole does not intersect with the floor of the vertical image or is not at least as wide as the door.

[0005] (5) Various embodiments described herein use a 3D depth scanner and a 2D camera working in tandem to create a 3D mesh.

[0006] (6) Various embodiments described herein capture a 2D virtual image of a 3D mesh by copying a slice of the 3D mesh with the y-axis of the 2D virtual image being the length of the 2D segment and the x-axis of the 2D virtual image being a value derived from the 3D mesh.

[0007] (7) In various described embodiments, slices of the 3D mesh are defined as one pixel wide.

[0008] (8) In various described embodiments, the X axis is a value derived from a 3D mesh, and includes finding the minimum value in the 3D mesh along a 2D segment.

[0009] (9) In various described embodiments, the X axis being a value derived from the 3D mesh further includes determining the maximum value in the 3D mesh along a 2D segment.

[0010] (10) Various embodiments described herein include locating faces in a 3D mesh whose normal vectors are opposite to gravity, and recording areas of faces in the 3D mesh whose normal vectors are opposite to gravity as up-facing areas.

[0011] (11) In various described embodiments, the faces in the 3D mesh whose normal vectors are aligned with gravity are located, and the areas of the faces in the 3D mesh whose normal vectors are aligned with gravity are recorded as downward facing areas.

[0012] (12) In various described embodiments, a histogram of the upward facing area at different heights is created, and a histogram of the downward facing area at different heights is created.

[0013] (13) In various described embodiments, the highest histogram height in the upward area is determined and declared to be the ceiling.

[0014] (14) In various described embodiments, the height of the highest histogram in the downward facing area is determined and this is declared as the defined floor.

[0015] (15) In various embodiments described, the X-axis is the height of the highest histogram in the up-facing area.

[0016] (16) In various described embodiments, the vertical image floor-intersecting holes include defined floor-intersecting holes.

[0017] (17) Various embodiments described herein relate to a system for locating a shape within a 3D mesh. The system includes a processor in communication with a memory that stores the 3D mesh. The processor is configured to locate a 2D segment correlated with a 3D mesh representing a surface of the room, capture a virtual 2D image of the 3D mesh along the 2D segment, find a hole in the virtual 2D image, find a position of the hole in the 3D mesh, offset the position of the hole in the 3D mesh to form a hole offset, cast a ray from a center of the hole offset into the 3D mesh, determine whether the cast ray hits a back side of a face in the 3D mesh, classify the hole as an error in the 3D mesh if the cast ray hits the back side of one or less faces in the 3D mesh, determine a floor of a vertical image if the cast ray hits the back side of more than one face in the 3D mesh, measure a width of the hole, classify the hole as a door if the hole intersects the floor of the vertical image and is at least as wide as a door, and classify the hole as a window if the hole does not intersect the floor of the vertical image or is not at least as wide as a door.

[0018] (18) In various embodiments described, six beams of light are emitted.

[0019] (19) In various described embodiments, six radiation beams are emitted in six cardinal directions.

[0020] (20) Various embodiments described herein relate to a non-transitory machine-readable storage medium encoded with instructions for execution by a processor to define holes and windows in a 3D mesh. The non-transitory machine-readable storage medium includes instructions for locating a 2D segment correlated with a 3D mesh representing a surface of a room, instructions for capturing a virtual 2D image of the 3D mesh along the 2D segment, instructions for finding a hole in the virtual 2D image, instructions for finding a position of the hole in the 3D mesh, instructions for offsetting the position of the hole in the 3D mesh to form a hole offset, instructions for casting a ray into the 3D mesh from the center of the hole offset, instructions for determining whether the cast ray hits the back side of a face in the 3D mesh, instructions for classifying the hole as an error in the 3D mesh when the cast ray hits the back side of one or less faces, instructions for determining a floor of a vertical image and further measuring a width of the hole when the cast ray hits the back side of more than one face, and instructions for classifying the hole as a door when the cast ray hits the back side of

[0021] (21) In various described embodiments, a 3D depth scanner and a 2D camera working in tandem are used to create a 3D mesh.

[0022] (22) In various described embodiments, the depth scanner is a LiDAR device.

[0023] (23) Various described embodiments include instructions to classify a hole as a window if the hole does not intersect the floor in the vertical view or if the hole is not at least as wide as the door.

[0024] (24) In various embodiments, it is perhaps worthwhile to further discuss the 2D segments correlated to the walls.

[0025] (twenty five) (26) In its broadest terms, this invention is about detecting features that lie within the 3D planes of a model. There are a variety of ways to extract or discover these 3D planes.

[0026] (27) (28) In this particular application, we use our prior invention, which helps extract 2D floor plans. Walls are represented in the plan view as 2D segments of polygons. However, we essentially convert these into 3D planes that coincide with the walls for this feature detection step.

[0027] (29) (30) For example, the present invention can also be used to detect skylights in ceilings instead of walls.

[0028] (31) For a better understanding of various example embodiments, reference is made to the accompanying drawings. [Brief explanation of the drawings]

[0029] [Figure 1] An example 3D mesh structure 100 is shown superimposed on an existing structure, where the mesh structure extends the classification of windows into the rooms behind them. [Figure 2] An example of an existing 3D mesh structure is shown, where the classification of doors extends to the interior of the glass structure. [Figure 3] 3 shows different views of the 3D mesh structure of FIG. 2. [Figure 4] An overview of an example method for correctly classifying windows, doors, and voids in 2D, 3D, and composite meshes is presented. [Figure 5] 1 shows an exemplary hardware diagram for implementing a controller / hardware device. [Figure 6] 1 illustrates an example of a distributed computing system that can be used in the embodiments described herein. [Figure 7A]FIG. 1 is a diagram of an example of the back of an exemplary scanner that can be used to develop 2D and 3D meshes that can be used with the implementations described herein. [Figure 7B] FIG. 1 is a diagram of an example of the front face of an exemplary scanner that can be used to develop 2D and 3D meshes that can be used with the implementations described herein. [Figure 8] An example of a method for detecting features in an AR environment is shown. [Figure 9] FIG. 1 is an example diagram of segments defining a room floor plan that can be used in the methods and systems presented herein. [Figure 10] FIG. 1 is an example of a mesh cloud that can be used to define a portion of a room in the methods and systems presented herein. [Figure 11a] It corresponds to the 3D mesh surrounding the surface of the room referenced by the segments in FIG. [Figure 11b] This corresponds to the virtual 2D image of the 3D mesh shown with reference to FIG. 11a. [Figure 12] It corresponds to a virtual 2D image of the room surface represented by the segment. [Figure 13] It corresponds to a virtual 2D image of the room surface represented by the segment. [Figure 14] It corresponds to a virtual 2D image of the room surface represented by the segment. [Figure 15] An example of a method for classifying located holes is shown below. [Figure 16] Offset holes are shown. [Figure 17] An example of a method for determining whether or not there is a hole on the ground will be described. DETAILED DESCRIPTION OF THE INVENTION

[0030] (48) The descriptions and drawings presented herein illustrate various principles. It will be appreciated that those skilled in the art will be able to conceive of various configurations that, while not explicitly described or shown herein, embody these principles and are included within the scope of the present disclosure. As used herein, the term "or" is intended to mean a non-exclusive "or" (i.e., and / or) unless otherwise indicated (e.g., "or else" or "or in the alternative"). In addition, the various embodiments described herein are not necessarily mutually exclusive and can be combined to create additional embodiments incorporating the principles described herein.

[0031] (49) Augmented reality (AR / LiDAR) allows for the capture of detailed clouds of 3D points in real time with reasonable precision and accuracy. A 3D point cloud or 3D mesh of an environment can be captured simultaneously with the same environment captured in a 2D augmented reality (AR) environment using a depth measurement system / depth scanner such as LiDAR. This cloud of 3D and 2D points can be structured to form a 3D mesh cloud. One application is detecting closed windows and doors within this 3D mesh cloud. However, such detection is not without error. One type of error is overinclusive classification, making it very difficult to use. For example, when the outline of a closed door or window is detected in 2D and then projected onto a 3D surface, or detected in 3D from scratch, the 2D door or window classification will project beyond the original surface and into the space behind it, which will be classified as a door or window rather than as part of the room behind it. Not only does this result in a false and inaccurate characterization of the portion of the space behind the door, but as the 3D perspective changes, so does what is "behind" an object classified as a door or window. This particular mischaracterization can occur when the projection and image points do not converge.

[0032] (50) Figure 1 shows an example 3D mesh structure 100 superimposed on an existing structure, where the mesh structure incorrectly extends the window classification into the room behind it. Window 105, defined as a window and its classification marked by a red mesh, may be seen erroneously extending into the left wall and ceiling 115 of room 110. Figure 2 shows an example of an existing 3D mesh structure 200, where the door classification (yellow mesh) 205, 210 incorrectly extends into the glass structure separated by columns 215, 220. Figure 3 shows the same 3D mesh structure 300 viewed from a different angle. Here, it can be seen that the incorrect yellow mesh door classification 310 has been significantly reduced, and most of the previous yellow mesh door structure has now been incorrectly classified as a window, as indicated by the red mesh 305. This unstable classification not only leads to problems with incorrect classification, but also to problems with classifications potentially changing unexpectedly and unjustifiably.

[0033] (51) Another problem is that currently, when holes are present in the generated mesh, they are very likely either not classified at all or misclassified. An example is the 3D mesh generated by a bookshelf facing a wall. A slice of the 3D mesh around this wall shows the hole in the image where the bookshelf projects into the room, but this hole is caused by the bookshelf obscuring the wall. The mesh is formed around the bookshelf, not the wall. This same phenomenon can be seen with the shelves shown at 1105a and 1110a in FIG. 11a. In the image of the wall shown at 1100b, holes 1105b and 1110b are present in the image because the 3D mesh was formed around the shelf, not the wall; therefore, slice 1100b shows the hole where the shelf projects into space.

[0034] (52) Until now, distinguishing between miscalculated holes and actual voids in a representative object has been a challenging image processing problem without a clear solution. Failure to correctly classify holes not only results in a noisy, unreliable surface containing many features that may not actually exist, but also leads to unreliability regarding the location of actual features such as windows, doors, and voids.

[0035] (53) To address these and other aspects of detection and classification within an AR framework, various embodiments described herein relate to highly accurate classification of objects that may appear as holes within 2D and 3D and mixed 2D / 3D frameworks.

[0036] (54) Generally speaking, the embodiments disclosed herein discuss detecting features within 3D planes of a model. There are various ways to extract these 3D planes. Figure 4 illustrates, at 410, one way to find 3D planes that may contain features to be classified. Other ways of extracting 3D planes are also possible.

[0037] (55) Figure 4 shows an overview 400 of an example method for correctly classifying features in 2D, 3D, and composite meshes. In various embodiments, method 400 can be performed by a processor 520 in a scanner, e.g., 630, a distributed system, a single processor, a single controller, multiple controllers in a distributed system, etc. Various alternative configurations and methods for detecting features will also be apparent.

[0038] (56) The method begins at step 405 and proceeds to 410, where an existing 3D mesh is divided into segments (which may be surface planes of a room, such as a seam between a wall and a ceiling). A segment represents a floor plan and has an associated room surface, such as a wall, ceiling, or floor, as described in more detail with reference to FIGS. 9-14. One of these segments is selected. Then, at step 415, a 2D image is captured from the 3D mesh version of the room surface represented by this segment at the location of the segment, as described in more detail with reference to FIGS. 11a and 11b. The 2D image is then examined to find features at operation 420. If a feature is located, it is classified at operation 425. The classification may be a window, door, hole, or other feature. At decision point 430, the 2D image is examined for additional holes. The process begins at operation 420 and repeats until no more holes are found, at which point it is determined at decision point 435 whether there are any more floor plan segments or surface planes. If so, the process begins at operation 410 and repeats until there are no more segments. The method then ends at operation 440.

[0039] (57) Figure 5 illustrates an exemplary hardware diagram 500 for implementing a controller / hardware device. As shown, device 500 includes a processor 520, memory 530, a user interface 540, a communication interface 550, and storage 560, which are interconnected by one or more system buses 510. It should be understood that Figure 5 is, in some respects, an abstraction, and the actual organization of the components of device 500 may be more complex than depicted.

[0040] (58) Processor 520 may be any hardware device capable of executing instructions stored in memory 530 or storage 560 or otherwise processing data. Thus, a processor may include a microprocessor, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a vector processor, or any other device capable of performing the logical functions described herein. In a multiprocessing system, multiple processing units execute machine-executable instructions to increase processing power. Thus, multiple processors, and multiple elements comprising a processor, may execute simultaneously. However, it should be appreciated that in various embodiments, elements belonging to processor 520 need not be physically co-located. For example, multiple processors may be mounted on boards that are physically separate from one another.

[0041] (59) Memory 530 may include various types of memory, such as, for example, L1, L2, or L3 cache or system memory. Accordingly, memory 530 may also include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read-only memory (ROM), or other similar memory devices. It will be appreciated that in embodiments in which the processor includes one or more ASICs (or other processing devices) and implements one or more of the functions described herein in hardware, software described as corresponding to such functions in other embodiments may be omitted.

[0042] (60) User interface 540 may include one or more devices that enable communication with a user, such as an administrator. For example, user interface 540 may include a display and a keyboard for receiving user commands. User interface 540 may also include a mouse. In some embodiments, user interface 540 may include a command line interface or a graphical user interface, which may be displayed on a remote terminal via communication interface 550.

[0043] (61) The communication interface 550 may also include one or more devices that enable communication with other hardware devices. For example, the communication interface 550 may include a network interface card (NIC) configured to communicate according to an Ethernet protocol. The communication interface 550 may also include a Bluetooth transmitter, receiver, antenna, and specialized control chip. In addition, the communication interface 550 may implement a TCP / IP stack for communication in accordance with the TCP / IP protocol. As will be apparent, the communication interface may also include various alternative or additional hardware or configurations for the communication interface 550.

[0044] (62) Storage 560 may include one or more machine-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, or similar storage media. In various embodiments, storage 560 may store instructions for execution by processor 520 or data that processor 520 may process. For example, storage 560 may store a basic operating system 561 to control various basic operations of hardware 500. Storage 560 may also store A / R software 563. A / R software 563 may be capable of developing a 3D mesh using 2D and 3D inputs. Storage 560 may also include classification software 565. Classification software 565 classifies objects within the 3D mesh, such as windows, doors, voids, and other objects that may be represented within the 3D mesh.

[0045] (63) It will be apparent that the various information described as being stored in storage 560 may additionally or alternatively be stored in memory 530. In this regard, memory 530 may be considered to constitute a "storage device," and storage 560 may be considered a "memory." Various other configurations will also be apparent. Furthermore, both memory 530 and storage 560 may be considered to be "non-transitory machine-readable media." As used herein, the term "non-transitory" shall be understood to include all forms of storage, including both volatile and non-volatile memory, but excluding transitory signals.

[0046] (64) While host device 500 is shown as including one of each of the described components, in various embodiments, various components may be duplicated. For example, processor 520 may include multiple microprocessors. These microprocessors may be configured to independently perform the methods described herein, or the multiple processors may be configured to cooperate to perform the steps or subroutines of the methods described herein to achieve the functionality described herein. Furthermore, when device 500 is implemented within a cloud computing system, the various hardware components may reside in separate physical systems. For example, processor 520 may include a first processor in a first server and a second processor in a second server.

[0047] (65) FIG. 6 illustrates an example of a distributed computing system 600 that can be used in the embodiments described herein. A controlled space 605 can have multiple controllers (e.g., 610, 615, 620) within the space. Each of these controllers can include some or all of the controller / hardware system illustrated with reference to 500 in FIG. 5. In some embodiments, the distributed computing system can run without access to the Internet 665. In some embodiments, the distributed computing system can also run as an edge computing system, with one or more edge devices allowing selective connection to the Internet through these. In some embodiments, the distributed computing system allows applications (e.g., software) to run on multiple controllers (e.g., any combination of 610, 615, 620) simultaneously, sequentially, or a combination of both. The same application can also run in concurrent processes on different processors, different controllers, etc., so that a user can view these controllers as a unified whole. This allows for redundancy so that if one component fails, another component can seamlessly take over. These distributed applications can be stored and run entirely within the controlled space 605 without accessing the external Internet 665. This allows for low latency because, for example, processing times are not affected by cloud / Internet delay times. In some cases, one or more controllers 610, 615, 620 can also connect to the Internet 665. This connection can be siloed off from the majority of computing power in the controlled space 605.

[0048] (66) A distributed computing system can be implemented as a client-server system, with one controller designated as the master server. A distributed system can also be implemented as a peer-to-peer network, use virtual servers, etc. An application can be divided into multiple pieces, e.g., partitioned among controllers. Controllers can self-federate, e.g., they self-assemble into a network. Controllers can vote to elect a leader. If the network is damaged and the current leader can no longer lead, a new leader can be selected by at least some of the surviving controllers. This provides built-in redundancy. The leader controller can determine how to divide the workload among the controllers. A scanner 630 (described in more detail with reference to Figures 7a and 7b) can also be included. This scanner can be used to construct 2D and 3D meshes.

[0049] (67) The controller can have one or more connectors (650) that connect to resources 635. The controller can have a connector that connects to modules 640 via module connectors 655. The modules can have resource connectors 660 that are wired directly to resources 645. The resources 635, 645 can be any type of resource that can be wired to the controller. For example, without limitation, the resources can be HVAC resources such as heating, cooling, and storage resources, entertainment resources such as a sound system and TV, lighting resources, safety resources such as door locks, etc. The controller 620 can control the resources 645, and vice versa, via the resource connectors 660 that communicate with the module connectors 655.

[0050] (68) This allows the controller to control resources, such as turning on a heater, by passing information through module 640 via module connector 655. It then sends a message to resource 645 via resource connector 660, such as telling it to turn on a heater. Resources 635 can be wired directly to controller 615 via one or more resource connectors 650. In some embodiments, resources can also be wired to one or more modules. In certain embodiments, the controller may not control a particular resource, but may infer its state from scanners, sensors, the state of other resources, etc. In some embodiments, controllers (e.g., 610, 615, 620) run without using the Internet 665. This significantly improves the security of the system because system 600 is immune to Internet attacks, such as DDoS attacks, zero-day threats, etc.

[0051] (69) 700A in FIG. 7A and 700B in FIG. 7B are the back and front views of an exemplary scanner, such as scanner 630. Scanner 630, in some described embodiments, can be used to scan and create a 3D mesh using both a 2D and a 3D scanner. This scanner can incorporate some or all of the elements described with reference to exemplary hardware diagram 500. That is, the system described in FIG. 5 can be implemented using a mobile phone 700a or other portable computing device having a 2D camera 705a and a depth scanner 710a. In some embodiments, the depth scanner can be a LiDAR device. In some embodiments, the system includes a user input device. This user input / output device can be a touchscreen 705b. In some embodiments, the 3D depth scanner and the 2D camera can work in conjunction to create a 3D mesh. Furthermore, there can be a correlation between the photographs taken by the 2D camera and the 3D mesh created by the depth scanner, such that locations found in the 2D camera photographs can correspond to locations in the 3D mesh.

[0052] (70) Figure 8 illustrates an example method 800 for detecting features in an AR environment. In various embodiments, method 800 can be performed by a processor 520 within a scanner, e.g., 630, a distributed system, a separate processor, a separate controller, etc. Various alternative configurations and methods for detecting features will also be apparent.

[0053] (71) The method begins in step 805 and proceeds to step 810, where a rough floor plan can be created using floor plan detection, such as a depth scanner. The depth scanner can be an augmented reality (AR / LiDAR) device. This detection can be capable of capturing a detailed 3D point cloud with reasonable precision and accuracy. Such detection can be performed in real time by walking through the building and surveying the area with a camera / depth scanner device, for example, as shown with reference to 630 in FIG. 6 and further shown with reference to FIGS. 7a and 7b. This 3D point cloud, augmented with 2D points, can be automatically structured to create a 3D mesh. An example of such a 3D point cloud can be seen at 1000 with reference to FIG. 10. Once the 3D mesh is created, a floor plan can be constructed for each room surface using a user-held depth scanner device having a display screen such as that described with reference to FIGS. 7a and 7b. In an embodiment, a user scans a room with a handheld scanner and places user marks on the display screen indicating the location of room surfaces. When new room surfaces are determined through the user marks, depth scanner calculations, and heuristics, they are presented to the user with their appearance in the current floor plan. If the floor plan shows errors, the user can place other marks to more precisely specify the room surface locations. Such a system and method for extracting a series of segments that can be understood to be room surfaces is described in U.S. Patent Application No. 17 / 459,084, filed September 15, 2021, which is incorporated herein by reference in its entirety for all purposes. To improve scans of incomplete floor plans, a user-held depth scanner device having a display screen, such as that described with reference to FIGS. 7a and 7b, can be used to construct a floor plan for each room surface. The user scans the room and places marks on the display screen indicating the location of the room surfaces.When a new room surface is determined by user marks, depth scanner calculations, and heuristics, the user is given its appearance in the current floor plan. If the floor plan shows an error, the user can place another mark to more precisely specify the room surface location. Such a system and method for extracting floor plans is described in U.S. Patent Application No. 17 / 855,513, filed June 30, 2022, which is incorporated herein by reference in its entirety for all purposes.

[0054] (72) In some embodiments, a 3D mesh can be cut into a plane and used instead of floor plan segments. For example, a ceiling may have a skylight that needs to be classified. In such a case, wall-ceiling segments can be used. In operation 815, a segmented floor plan is created using the accurate floor plan described with reference to operation 810. To do this, a path through the graph of segments must be determined that passes through each vertex of the segment exactly once. In operation 820, a segment is selected. FIG. 9 is a diagram of an example of four 2D segments 905, 910, 915, and 920 that define a floor plan of a room 900. These can be used in the methods and systems presented herein, such as those described and created with reference to 810 and 815 of FIG. 8. FIG. 10 is an example of a mesh cloud 1000 that defines a portion of a room, such as room 900. This mesh cloud can also be described as a virtual world captured by a 3D depth sensor. In an embodiment, the mesh is made up of vertices, also known as faces, that have normal vectors that are aligned with gravity or the opposite direction of gravity. Figure 11a corresponds to a 3D mesh that encloses the surface of the room referenced by segment 905. Figure 11b corresponds to a virtual 2D image 1100b of mesh 1100a represented by 2D segment 905. Figure 12 corresponds to a virtual 2D image 1200 of the surface of the room represented by segment 910. Figure 13 corresponds to a virtual 2D image 1300 of the surface of the room represented by 2D segment 915, and Figure 14 corresponds to a virtual 2D image 1400 of the surface of the room represented by 2D segment 920. 2D segment 905 may be selected as an example of a segment selected in operation 820.

[0055] (73) In operation 825, a virtual 2D image is captured from a mesh around the selected plane, as shown with reference to FIG. 11b and the text thereunder. In one embodiment, the length of the virtual 2D image is the same as the length of the plane, but the height of the virtual 2D image may not be known. In such cases, various embodiments can be used to determine the height. One such embodiment is shown with reference to FIG. 17 and the text thereunder. The virtual 2D image may also have a width. In one embodiment, the width is in meters, feet, or some other width. In other embodiments, the virtual 2D image may have a different width.

[0056] (74) Figure 11a is a mesh 3D representation of a room surface 1100a, showing representations of wall cabinets, designated 1105a, 1110a. Figure 11b is a virtual 2D image of this mesh representation 1100a. The 3D shelves 1105a, 1110a now appear at least partially as holes 1105b, 1110b. This is likely because the mesh is formed around the outside of the shelves, leaving holes at the locations of the holes in the virtual 2D image.

[0057] (75) At operation 830, holes are found. In one embodiment, region segmentation is used. In one embodiment, edge detection is used to find holes. In one embodiment, the virtual 2D image is systematically searched to find locations where there are pixels with values ​​representing voids, i.e., values ​​indicating holes. In one embodiment, a window or door (which may be closed) may have been detected in different ways. This value may be a range of dark values ​​or a different range of values. In one embodiment, a minimum number of correlated pixels may be required to define a hole. In one embodiment, multiple hole detection methods may be used. A window or door (which may be closed) may have been detected in different ways, but may now be redefined as a hole. At operation 835, holes are classified. Hole classification is described in more detail with reference to FIGS. 15 and 16. At decision point 840, the virtual 2D image of the selected segment is repeatedly checked to see if there are any more holes within the segment. If more holes are found, the method resumes at operation 830. If there are no more holes in this particular segment, then at decision point 845 it is determined whether there are more segments to check for holes. If there are more segments, then the method repeatedly begins again at operation 820. If there are no more segments, then at 850 the method ends.

[0058] (76) Figure 15 illustrates an example method 1500 for classifying located holes, such as in an AR environment. In various embodiments, method 1500 can be performed by a processor 520 within a scanner, e.g., 630, a distributed system, a separate processor, a separate controller, etc. Various alternative configurations and methods for classifying located holes will also be apparent.

[0059] (77) The method begins at operation 1505 and then proceeds to operation 1510, where the located 2D hole is offset within the 3D mesh from which the virtual image was captured, e.g., as shown with reference to FIG. 11a. This may be offset in a perpendicular direction from the portion of the mesh corresponding to that segment, e.g., perpendicularly from the direction of the floor or ceiling. In some embodiments, the offset itself is two-dimensional, and in some embodiments, the offset is three-dimensional.

[0060] (78) FIG. 16 shows a side view of an offset hole 1600, e.g., hole 1005, offset 1610 at a perpendicular angle from a floor 1620. Alternate embodiments may have the offset in a different direction than the floor or ceiling. In operation 1515, a location within the offset is defined. In one embodiment, a center 1615 of the offset area is defined. The offset from hole 1610 may be the same length 1605 as the hole itself, the offset may be a set offset, or the offset may be a different length. Once the center of the offset is defined, in operation 1510, rays are emitted in multiple directions from the center of the offset area. For example, rays may be emitted 1520 in six cardinal directions 1625, 1630, 1635, 1640, 1645, 1650, e.g., cardinal directions in 3D space. In some embodiments, fewer or more than six rays are emitted. In some embodiments, rays are cast in directions other than the cardinal directions. In some embodiments, rays are cast long enough to intersect with features on the room surface they represent, if any. For example, a ray emitted from hole 1105b must be long enough to hit the shelf shown in 1105a. The ray must not be long enough to hit a distant room surface. At decision point 1525, the number of rays hitting the back side of a face in the mesh is calculated. Front-facing intersections are ignored. If there are zero or one rays hitting the back side of a face in the 3D mesh, then in operation 1530, the hole is classified as an artifact. Because a single ray could hit the ceiling or floor of the artifact, this does not represent an actual hole, but indicates an error in the mesh. The method then ends at operation 1535. At decision point 1535, if more than one ray hits the back side of the 3D mesh surface, the apparent hole is determined to be a real hole, and therefore classification is required. The method proceeds to operation 1540, where the floor of the virtual 2D image is determined.In one embodiment, a door is naturally assumed to start at floor level, which is used to determine whether the hole is a door. One embodiment for determining the floor is described with reference to FIG. 17 and the accompanying text. In one embodiment, the floor determination may be made in advance. In other embodiments, different assumptions about the door's placement may be made. At decision point 1545, it is determined whether one of the hole's boundaries is on the floor or close enough to the floor. This may be done by checking the hole's lowest Z value against a determined floor Z value (as described with reference to FIG. 17). An allowable deviation from the floor Z level may be established so that holes that are a predetermined amount higher than the floor Z value are still considered "on the floor." If the hole is not at floor level, then operation 1565 classifies the hole as a window. An example of a window in the virtual 2D image can be seen at 1210 in FIG. 12. The window classification may include other parameters, such as size, orientation, etc., to determine whether the hole is a window. If, at decision point 1545, it is determined that one of the boundaries of the hole is on the floor, then at decision point 1550 it is determined whether the hole is wide enough to be a door, i.e., whether it is the width of a door. The determination of how wide a door must be may be made by a constant or by an analysis that takes into account the location of the hole, the type of room it is in, etc. If the hole is wide enough, then at operation 1555 it is classified as a door, and then at operation 1560 the method ends. In some embodiments, the hole may also be checked to see if it is tall enough to be a door. In some embodiments, other parameters are included. Figure 12 shows an example of a hole that can be classified as a door at 1205. It will be noted that the hole is on or close enough to the ground, and is tall and wide enough to be a door. If at decision point 1550 it is determined that the hole is not wide enough to be a door, then at operation 1565 the hole is classified as a window and at operation 1570 the method ends.

[0061] (79) FIG. 17 illustrates an example method 1700 for determining whether a hole is at ground level or in a ceiling. This may be used, for example, in operation 825 of FIG. 8 to determine the y-value of a virtual image captured within a mesh. It may also be used, for example, to classify doors or windows, as described with reference to FIG. 1 and the text thereunder. In various embodiments, method 1700 may be performed by a processor 520 within a scanner, e.g., 630, a distributed system, a separate processor, a separate controller, etc. Various alternative configurations and methods for determining whether a hole is located at ground level will also be apparent.

[0062] (80) The method begins at operation 1705 and then proceeds to operation 1710, where a 3D mesh representing the room (e.g., FIG. 9) capturing the segment containing the hole is used to determine minimum and maximum Z values ​​(see, e.g., 820 in FIG. 8 and the text thereunder). The faces of this 3D mesh are oriented toward gravity, with some faces facing upward (Z+) and some faces facing downward (Z-). At operation 1715, individual faces in the mesh that face upward, e.g., have a Z+ value between the minimum and maximum Z values, are identified. At operation 1720, the areas of these faces found at operation 1715 and their Z values ​​are recorded. At operation 1725, faces in the mesh that face downward (face normal vectors aligned with gravity) that are between the minimum and maximum Z values ​​are identified. At operation 1730, the areas of the faces found at operation 1715 and their Z values ​​are recorded. At operation 1735, a histogram of area by Z value is created. The number of bars in the histogram can be determined in any number of ways. There can be a fixed number of bars, the number of bars can be related to the size of the maximum and minimum Z values, etc. At operation 1740, the tallest bars (e.g., the bars with the widest area) are determined in the upward (Z+) and downward (Z-) directions. At operation 1745, the histogram bar with the highest Z+ value is determined to be the ceiling. Similarly, the histogram bar with the highest Z- value is determined to be the floor. This can be thought of conceptually as capturing the ceiling and floor in the mesh, but not the space within the room, which can typically be thought of as white space. The method then ends at operation 1750.

[0063] (81) From the foregoing description, it should be apparent that various example embodiments of the present invention may also be implemented in hardware or firmware. Furthermore, various example embodiments may also be implemented as instructions stored on a non-transitory machine-readable storage medium, which, when read and executed by at least one processor, may perform the operations detailed herein. A non-transitory machine-readable storage medium includes any mechanism for storing information in a form readable by a machine, such as a personal or laptop computer, a server, or other computing device. Thus, a non-transitory machine-readable storage medium may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, and similar storage media.

[0064] (82) It should be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the present invention. Likewise, it should be appreciated that any flow charts, flowcharts, or the like represent various processes substantially embodied in a machine-readable medium and thus capable of being executed by a computer or processor, whether or not such a computer or processor is explicitly shown.

[0065] (83) While various exemplary embodiments have been described in detail above, with particular reference to certain exemplary aspects, it should be understood that the invention is capable of other embodiments and that its details are susceptible to modifications in various obvious respects. As will be readily apparent to those skilled in the art, variations and modifications can be made while remaining within the spirit and scope of the invention. Accordingly, the foregoing disclosure, description, and figures are for illustrative purposes only and in no way limit the invention, which is defined solely by the claims.

Claims

1. 1. A processor-implemented method for locating features within a 3D mesh, comprising: locating, in a memory associated with said processor, 2D segments correlated with a 3D mesh representing the surfaces of the room; determining a virtual image bottom edge and a virtual image top edge of the 3D mesh; capturing a virtual 2D image of the 3D mesh along the 2D segment using the virtual image bottom edge and the virtual image top edge; Finding holes in the virtual 2D image; b) measuring the width of the hole; c) determining a structural characteristic of the hole when the hole intersects with the bottom edge of the vertical image and when the hole is at least as wide as a predetermined door; A method comprising:

2. 2. The method of claim 1, wherein the hole is classified as a window when the hole does not intersect the bottom edge of the vertical image or when the hole is not at least as wide as the predetermined door.

3. The method of claim 1 , wherein the 3D mesh is created using a 3D depth scanner and a 2D camera working in conjunction.

4. The method of claim 1 , further comprising the step of determining whether the hole is an artifact.

5. 5. The method of claim 4, wherein the step of determining when the hole is not an artifact comprises: Finding the location of the hole in the 3D mesh; offsetting the positions of the holes within the 3D mesh to form hole offsets; casting a ray from the center of the hole offset into the 3D mesh; determining that the hole is not an artifact if the radiation strikes the back side of more than one surface; A method comprising:

6. The method of claim 1 , wherein the surface of the room comprises a floor, a ceiling, or a wall.

7. 2. The method of claim 1, wherein capturing a virtual 2D image of the 3D mesh along the 2D segment comprises capturing the virtual image with a y-axis extending from a bottom edge of the virtual image to a top edge of the virtual image.

8. 2. The method of claim 1, wherein the step of determining the bottom edge and top edge of the virtual image of the 3D mesh includes determining the minimum Z value of the room as the bottom edge of the virtual image and the maximum Z value of the 3D mesh as the top edge of the virtual image, and further includes the steps of finding upward-facing surfaces and recording the area of ​​the upward-facing surfaces, and finding downward-facing surfaces and recording the area of ​​the downward-facing surfaces.

9. 8. The method of claim 7, further comprising the steps of: locating faces in the 3D mesh whose normal vectors are opposite to gravity; recording areas of the 3D mesh whose normal vectors are opposite to gravity as upward facing areas; locating faces in the 3D mesh whose normal vectors are aligned with gravity; recording areas of the surfaces in the 3D mesh whose normal vectors are aligned with gravity as downward facing areas; A method comprising:

10. 10. The method of claim 9, further comprising the steps of: creating a histogram of the upward facing area at different heights; and creating a histogram of the downward facing area at different heights.

11. 10. The method of claim 9, further comprising the step of determining the highest histogram height in the downward facing area and declaring it as the defined floor.

12. 1. A system for locating features in a 3D mesh, comprising: a processor in communication with a memory storing a 3D mesh; the processor: locating in said memory 2D segments correlated with a 3D mesh representing the surface of the room; determining a bottom edge of a virtual image of the 3D mesh and a top edge of the virtual image; capturing a virtual 2D image of the 3D mesh along the 2D segment using the virtual image bottom edge and the virtual image top edge; capturing a virtual 2D image of the 3D mesh along the 2D segment; Finding a hole in the virtual 2D image; Measure the width of the hole; If the hole intersects with the bottom edge of the vertical image and the width of the hole is equal to or greater than the width of the door, classify the hole as a door. The system is configured as follows:

13. 13. The system of claim 12, further comprising the act of determining when the hole is not an artifact.

14. 14. The system of claim 13, wherein the act of determining when the hole is not an artifact comprises: finding the location of the holes in the 3D mesh; offsetting the positions of the holes within the 3D mesh to form hole offsets; emitting a ray from the center of the hole offset into the 3D mesh; determining that the hole is not an artifact if the radiation strikes the back side of more than one surface; Including, the system.

15. 15. The system of claim 14, wherein the act of emitting a beam of light comprises an act of emitting six beams of light.

16. 16. The system of claim 15, wherein the six light beams are emitted in six cardinal directions.

17. 1. A non-transitory machine-readable storage medium encoded with instructions that a processor executes to locate features within a 3D mesh, comprising: instructions for locating, in a memory associated with said processor, 2D segments correlated by said processor with a 3D mesh representing a surface of a room; instructions to define a virtual image bottom edge and a virtual image top edge of the 3D mesh; instructions to capture a virtual 2D image of the 3D mesh along the 2D segment using the virtual image bottom edge and the virtual image top edge; instructions to find holes in the virtual 2D image; instructions to measure the width of the hole; instructions to classify the hole as a door when the hole intersects the bottom edge of the vertical image and when the hole is at least as wide as a door; A non-transitory machine-readable storage medium, including:

18. 20. The non-transitory machine-readable storage medium of claim 17, further comprising instructions for determining when the hole is not an artifact.

19. 20. The non-transitory machine-readable storage medium of claim 18, wherein instructions for determining when the hole is not an artifact comprise: instructions to find the location of the holes in the 3D mesh; instructions to offset the position of the hole within the 3D mesh to form a hole offset; instructions to cast a ray from the center of the hole offset into the 3D mesh; instructions for determining that the hole is not an artifact when the radiation strikes the back side of one or less surfaces; instructions for determining that the hole is an artifact when the radiation strikes the back side of more than one surface; A non-transitory machine-readable storage medium, including:

20. 20. The non-transitory machine-readable storage medium of claim 19, further comprising instructions for classifying the hole as a window when the hole does not intersect with a bottom edge of the vertical image or when the hole is not at least as wide as a door.

Images