Spatial Awareness Tool System

Abstract

The system described in the present disclosure can be used in a construction setting to facilitate the tasks being performed. The positions of the projector and augmented reality headset can be calculated and used to determine the images to be displayed to the worker based on a map of the work to be performed, such as a construction plan. The worker can use spatial awareness tools to make different positions perpendicular, horizontal, or equidistant from other positions. If the tools are close to the protected object, the power for the tools can be deactivated.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to U.S. Provisional Application No. 62 / 753,389 of the same title filed on October 31, 2018, the entire contents of which are incorporated herein by reference. Background of the Invention

[0004] The present disclosure relates generally to the field of construction tools. Construction work often requires non-digital hand tools that do not calculate their position, such as tape measures, levels, and power tools. Workers refer to construction plans, such as blueprints, to identify the work that needs to be performed. Summary of the invention

[0005] In general, embodiments of the present disclosure relate to tracking a physical location of a spatial awareness tool, associating the physical location with a construction plan, generating augmented data based on the association, and providing the augmented data to a user. A projector displays an image based on the augmented data representing the construction plan on the floor and walls of a room where work is to be performed. The projector positions virtual objects in the image at physical locations where physical objects would be positioned according to the construction plan.

[0006] The projector includes a switch key to allow the worker to switch the displayed image to a schematic diagram of the construction plan. A worker familiar with reading construction plans can view the projected image in the room and immediately understand which work must be performed in which parts of the room.

[0007] When generating the image based on the augmented data, the projector identifies the physical landscape of the wall, ceiling(s), and floor(s) on which the image is to be projected. The projector then generates the image based on the relationship of the physical landscape available to the projector to which objects in the construction plan are to be installed in / on the physical landscape.

[0008] An augmented reality (AR) headset receives the augmented data and generates an augmented image for display to a worker wearing the AR headset. The augmented image includes a physical image in front of the AR headset and the worker combined with virtual images of objects from the construction plan. The images of the virtual objects are placed in the augmented image at locations corresponding to where corresponding physical objects should be placed according to the construction plan.

[0009] In one embodiment, the location of the spatially aware drill is compared to a map of vulnerable or dangerous objects, such as electrical wires or water pipes that should be protected. The location of the protected object is based on the construction plan, or it can be based on an as-built model that reflects where the object is actually installed. The distance between the drill tip and the protected object is monitored and compared to a threshold. When the distance is less than the threshold, power to the drill is disabled, and the drill generates a signal to show the worker why the power to the drill is disabled.

[0010] In one embodiment, a spatially aware tool, such as a drill, is used to define an artificial reference frame. First, the position of the drill tip is calibrated, such as by placing the drill tip against a spatial tracking sensor, thereby calibrating the components of the drill to the room. The tracked position of the drill tip is placed at the origin of the artificial reference frame, such as a corner between two walls and a floor, and a signal is generated to identify the origin. The drill tip is then placed at another corner between one of the walls and the floor, and a second signal is generated to identify a first axis. The first axis is defined by connecting to the identified point.

[0011] The worker identifies special points in the spatially aware room relative to the artificial reference frame, such as the locations of drill holes. The artificial reference frame and these special points are transmitted to another worker using an AR headset. The second worker's AR headset displays an augmented image showing these special points indicated against the wall. Thus, the second worker can visualize these special points and the locations of the drill(s).

[0012] The physical location of the spatial awareness tool with the first worker is tracked by the system and sent to the remote worker. The remote worker views a virtual image of the construction plan at the physical location. The spatial awareness tool may include a camera that captures a physical image that is streamed to the remote worker. The physical image may be combined with the virtual image to create an augmented image. The augmented image is displayed to the second worker, such as via an AR headset or a projector.

[0013] In one embodiment, a menu of virtual items that can be installed is shown to a worker wearing an AR headset. The worker selects from different categories of items (such as HVAC (heating, ventilation, and air conditioning) or plumbing systems) to install. The worker selects the components to be installed in the room from the menu. The worker virtually installs the selected components by indicating the location of the selected components. The worker continues to select components and install them in and around the room. After the virtual construction is completed, the worker can indicate that the selected components need to be ordered and delivered to the room. In some cases, actual components can be manufactured remotely based on the virtual installation and delivered to the room for installation.

[0014] In a number of different embodiments, the system can incorporate a projector on a tripod, AR flashlight, light stand, sensor block, or pole, all of which are spatially aware and therefore system-enabled. These devices can work in conjunction with each other to identify points where work should be performed. These devices can rely on each other to determine their positions. The position determination of the device can be a corresponding position relative to an artificial reference frame, or it can be an absolute position relative to a construction plan. When these devices determine their positions based on other devices, they can jump across areas, providing spatially aware guidance for the area.

[0015] Additional features and advantages will be set forth in the following detailed description and will in part be apparent to those skilled in the art from the description or by practicing the embodiments as described in the written description and claims thereof and in the accompanying drawings. It should be understood that both the above general description and the following detailed description are exemplary.

[0016] The accompanying drawings are included to provide a further understanding and are incorporated in and constitute a part of this specification. The drawings illustrate one or more embodiments and together with the description serve to explain principles and operation of various embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a top view of a room implementing the system with a light stand, a projector, and an AR headset according to an embodiment.

[0018] Figure 2 According to the embodiment Figure 1 A perspective view of the room in which the system is implemented, with the projector showing the perspective-based augmented reality.

[0019] Figure 3 According to the embodiment Figure 1 A perspective view of the room where the system is implemented, with a projector showing schematic-based augmented reality.

[0020] Figure 4 is an augmented reality perspective view of a room implementing the system with an AR headset and two light stands according to an embodiment.

[0021] Figure 5 is an augmented reality perspective view of a room implementing the system with two light stands and an artificial reference frame on one wall according to an embodiment.

[0022] Figure 6 is a virtual perspective view of a room implementing the system according to an embodiment.

[0023] Figure 7 According to the embodiment Figure 6Perspective view of the room where the system is implemented.

[0024] Figure 8 According to the embodiment Figure 6 An augmented reality rendering of the room where the system is implemented.

[0025] Fig. 9 is an augmented reality perspective view of a room implementing a system identifying and displaying multiple locations according to an embodiment.

[0026] Fig.10 is an augmented reality perspective view of a room implementing a system that identifies and displays windows and multiple locations according to an embodiment.

[0027] Fig.11 is an augmented reality perspective view of a room implementing the system in which a plurality of artificial objects are identified and displayed according to an embodiment.

[0028] Fig.12 is a perspective view of a tripod implementing the system with a projector according to an embodiment.

[0029] Fig.13 is a perspective view of a flashlight implementing the system according to an embodiment.

[0030] Fig.14 is a perspective view of a light stand implementing the system according to an embodiment.

[0031] Fig.15 is a perspective view of a sensor block implementing the system according to an embodiment.

[0032] Fig.16 is a perspective view of an augmented reality (AR) headset implementing the system according to an embodiment.

[0033] Fig.17 is a perspective view of a benchmark for implementing a system according to an embodiment.

[0034] Fig.18 is a perspective view of a robotic total station system according to an embodiment.

[0035] Fig.19 is a perspective view of a robotic total station system according to an embodiment.

[0036] Fig. 20 is a perspective view of a tripod with an articulated laser according to an embodiment.

[0037] Fig.21 is a perspective view of a laser receiver having a vertical line laser according to an embodiment.

[0038] Fig. 22 is a perspective view of a multi-directional camera system according to an embodiment.

[0039] FIG. 23A to FIG. 23C is a series of overhead views of a room with a tripod having articulating lasers synchronized by interaction with a robotic station, according to an embodiment.

[0040] FIG. 24A to FIG. 24D is a series of overhead views of a room with a tripod having an articulating laser and beacon, according to an embodiment.

[0041] FIG. 25A to FIG. 25C is a series of top views of a room with a light table and a plurality of sensors according to an embodiment.

[0042] FIG. 26A to FIG. 26C is a series of top views of a room with a personal sensor and multiple light stands, according to an embodiment.

[0043] FIG. 27A to FIG. 27C is a series of top views of a room with a personal sensor and multiple infrared emitters according to an embodiment.

[0044] FIG. 28A to FIG. 28E is a series of top views of a room with a digital light table and multiple infrared receivers according to an embodiment.

[0045] FIG. 29A to FIG. 29B is a series of overhead views of a room with a non-continuously refreshing position detection projector according to an embodiment.

[0046] FIG. 30A to FIG. 30B is a series of overhead views of a room with a continuously refreshing position-detecting flashlight projector according to an embodiment.

[0047] Fig.31 is a top view of a room with multiple light tables, known locations, and personal sensors, according to an embodiment.

[0048] FIG. 32A to FIG. 32F is a series of overhead views of a room with sensors hopping across the room, according to an embodiment.

[0049] FIG. 33A to FIG. 33D is a series of overhead views of a room with a light stand that jumps across the room, according to an embodiment.

[0050] Fig.34 is an augmented reality perspective view of a room with an AR headset including a stereo camera according to an embodiment.

[0051] Fig.35 is an augmented reality perspective view of a completed object in a room according to an embodiment. DETAILED DESCRIPTION

[0052] Referring generally to the drawings, a number of different embodiments of using spatial awareness tools are shown. The systems described in this disclosure can be used in a construction setting to greatly facilitate the tasks being performed.

[0053] In a room where the system can calculate the location of tools and displays, the system provides images based on the construction plan to help workers identify where different parts or objects are to be installed. The system displays these images using projectors that display the images on the walls, floor, and ceiling of the room, or the system can provide these images through an augmented reality (AR) headset. The images provided can be actual images of the objects to be built and installed, or the images can be representative symbols such as can be found in blueprints, which allow workers to quickly understand the construction plan in the image being projected.

[0054] The system provides more accurate measurements than can be obtained more easily by using non-spatial awareness tools (such as hand tools). Workers can use spatial awareness tools in place of levels and tape measures. Spatial awareness tools (such as drills or AR headsets) provide feedback to the user about the different points being identified. Based on this information, workers can make these points perpendicular, horizontal, equidistant from other points, or any other arrangement that would otherwise require a level or tape measure.

[0055] The supervisor can identify a wall as a local artificial reference system and identify points on the wall where work will be performed. This identification of points can be performed by the supervisor without reference to the construction plan. These points can be saved to the system and another worker can walk through these rooms and perform the work identified by the supervisor by relying on the local artificial reference system(s).

[0056] The system can deactivate tools before they cause damage. For example, if a drill is about to bore a hole in a fragile or dangerous object (e.g., electrical wiring, water pipes), the system can deactivate power to the drill motor that rotates the drill and prevents damage to the protected object, but maintain power to the remaining systems and the drill, allowing the user to receive a message or indication communicating why the drill stopped.

[0057] refer to Figures 1 to 3, a tracking system (such as a light-based system) is shown according to an exemplary embodiment. System 40 includes a light stand 42, a projector 44, and an augmented reality (AR) headset 46. The light stand 42 emits a light pattern throughout the room, which can be used by receivers 120 on device 70 to determine their location. Device 70 includes spatial awareness devices in the room where the system is implemented, such as light stand 42, projector 44, AR headset 46, sensor 48, tool 72, tripod 110, flashlight projector 116, pole 122, robotic total station 140, articulated laser 142, laser receiver 146 with vertical line laser, and multi-directional camera system 148.

[0058] The projector 44 and AR headset 46 include one or more receivers 120 to determine their position and orientation, collectively referred to as position 52. The projector 44 and AR headset 46 use their position and orientation information to determine an image 54 to display for the user.

[0059] In various embodiments, the device 70 determines its position based on detecting light from the light stand 42. In alternative embodiments, one or more computer processors of the system 40 can determine the position of the device 70. This position determination analysis can be supplemented by an internal measurement unit (IMU) (such as a dynamic sensor including an accelerometer). The light stand 42 initially emits a pulse of light and then emits a series of light bands (such as calibration signal 150) in a pattern throughout the room. The device 70 detects the bands and measures the time period between the initial light pulse and the detected bands. Based on the length of the measured time period and the known position 52 of the light stand 22, the device 70 determines the first relative position 50 of the device 70 relative to the first light stand 42. The device 70 similarly determines the second relative position 50 of the device 70 relative to the second light stand 42. Based on the combination of the relative positions 50, the device 70 calculates its position 52 based on the first relative position 50 and the second relative position 50. In various embodiments, the device 70 calculates the orientation while calculating the position of the device 70. In various embodiments, the relative position 50 and the triangular position 50 include both the position and orientation of the device 70.

[0060] In another embodiment, the light stand 42 emits light encoded with timing data (e.g., the wavelength of the light corresponds to the time of light emission), and the device 70 decodes the received light to determine its position 50. In another embodiment, the light stand 42 emits a first light beam that rotates around the light stand 42 in a first rotational direction, and the light stand 42 also emits a second light beam that rotates around the light stand 42 in a second rotational direction opposite to the first rotational direction. In another embodiment, the light stand 42 emits a first light beam that rotates around the light stand 42 in a first rotational direction at a first speed, and the light stand 42 also emits a second light beam that rotates around the light stand 42 in the same rotational direction but at a second speed different from the first speed. By timing the difference between the times when the different light beams are received by the device 70, the device 70 can calculate its position 52. In another embodiment, the device 70 can determine its position 52 based on simultaneous localization and mapping (SLAM), where the device 70 calculates its position while simultaneously mapping the area in which the device 70 is located (e.g., via stereo vision, triangulation, LiDAR, and / or structured light projection). In alternative embodiments, other suitable methods for determining the three-dimensional position of the device 70 are used.

[0061] The projector 44 uses the receiver 120 to calculate its position 52, and stores the construction plan 56 to compare with the position 52 to determine the image 54 to be displayed. The projector 44 determines the image 54 by identifying the position of the projector 44 in the construction plan 56. The projector 44 uses the position 52 to identify the landscape 58 on which the projector 44 will display the image 54. The landscape 58 is the portion of the surface of the room, such as a wall, ceiling, or floor, where the projector 44 will display the image 54. Based on the identification of the landscape 58, the projector 44 generates the image 54 from the construction plan 56 to be displayed on the landscape 58. The projector 44 creates the object 60 in the image 54 so that the position of the object 60 corresponds to the position where the physical object 62 will be placed or constructed. Therefore, the user can view the image 54 displayed by the projector 44 and quickly understand and even see the position where the physical object 62 corresponding to the object 60 will be constructed. The projector 44 can be trained to receive an indication of the position 52 of the corner C within the room R, which can be particularly useful in the case where the room does not have a standard rectangular layout.

[0062] Image 54 may include object 60 ( Figure 2 ), or the image 54 may include a schematic diagram 64 of the construction plan 56 ( Figure 3 ). The image 54 may also include a digital model of an object known to be in view, or a scanned representation of a detected object.

[0063] The schematic diagram 64 includes hidden objects 66 that indicate where the physical objects 62 will be located behind the surface of the wall W or floor F. The schematic diagram 64 also includes points 68 where work needs to be performed, such as points where holes need to be drilled in the wall W or floor F.

[0064] Projector 44 may include a switch (e.g., a button) for switching between different types of images 54 being displayed. Projector 44 displays images 54 including objects 60 representing physical objects 62 to be installed according to construction plan 56 ( Figure 2 ), the projector 44 displays an image 54 ( Figure 3 ), or the projector 44 displays an image 54 including an outline of a hidden object 66 that has been installed or is to be installed in the floor or wall ( Figure 4 ).

[0065] Go to Figure 4 , the system 40 can display augmented reality (AR) by using an AR headset 46. The AR headset 46 can display an augmented image 78. The augmented image 78 includes a physical image 74 that is an actual view of the room or environment and a virtual object 60. In one embodiment, the AR headset 46 shows the user everything the user sees (e.g., a screen with an image of the real world overlaid with an augmented image). In another embodiment, the AR headset 46 includes a see-through portion where the user directly views the real world, and the see-through portion is a transparent display (e.g., an OLED display) with the augmented image 78.

[0066] Virtual objects 60 include sinks, doors, windows, and light switches, among other items to be installed in room R. Similar to projector 44, AR headset 46 determines its 3-D position 52 relative to construction plan 56, including its orientation. System 10 then calculates landscape 58 on which augmented image 78 will be displayed. In one embodiment, AR headset 46 performs some or all of the calculations, and in another embodiment, a remote computing device performs some or all of the calculations and streams the results and / or images to AR headset 46. Based on the location and area of ​​landscape 58, AR headset 46 identifies which objects 60 in construction plan 56 are within landscape 58, and therefore which objects 60 should be shown in image 54. After identifying objects 60 that will be in image 54, AR headset 46 renders representation 76 of object 60 to add to physical image 74, thereby creating augmented image 78. AR headset 46 also optionally renders floor representation 80 of object 60 to add to physical image 74. The floor representation 80 may indicate how the objects 60 will interact with the room R, such as by indicating the area that the door object 60 will open, or by the location in the wall W where the window object 60 will be placed.

[0067] Go to Figures 5 and 6 , the system 40 can display the augmented image 78 without reference to the construction plan 56. For example, a worker creates an artificial reference frame 100 (such as a Cartesian two-dimensional grid) and relies on the reference frame 100 without reference to the construction plan 56. The worker calibrates a tool 72 such as a drill that implements the system by first calibrating the drill tip 82, which will be used to define the artificial reference frame 100.

[0068] In one embodiment, the drill tip 82 is calibrated by performing several steps. First, the tool 72 is calibrated to an environment such as a room R. As a result of the calibration, the 3-D position 52 (including orientation) of the tool 72 within the room R is known by the system 40. However, the position 52 of the drill tip 82 (or a portion and / or end of the pole 122) may not be known with a sufficient level of accuracy because the length and positioning of the drill bit 84 within the tool 72 may not be known (e.g., the drill bit 84 may only be partially inserted into the key of the tool 72, the drill bit 84 may have a non-standard length, and drill bits 84 of different sizes or brands may have different lengths). To account for this variability in the position 52 of the drill tip 82 relative to the tool 72, the sensor 48 is placed against the drill tip 82. The system 40 calculates a differential position 88 of the drill tip 82 relative to the position 52 of the tool 72. Subsequently, the system 40 may combine the position 52 of the tool 72 with the differential position 88 of the drill tip 82 to calculate the position 52 of the drill tip 82.

[0069] Once the position 52 of the drill bit tip 82 is calibrated, the system 40 can remotely deactivate the tool 72 to prevent damage. For example, if the drill bit tip 82 position 52 is approaching a fragile or dangerous object (e.g., electrical wires, water pipes), the system 40 can deactivate power to the tool 72 before the drill bit tip 82 is closer to the physical object 62 than a threshold distance.

[0070] Once the drill tip 82 is calibrated, the worker creates an artificial reference frame 100. First, the worker moves the drill tip 82 to a floor corner at the intersection between the target wall TW, the side wall W, and the floor F on which the artificial reference frame 100 is arranged. The worker signals the system 40 that the drill tip 82 is at a first calibration point 102. The worker can signal the system by, for example, switching a button on the tool 72. The first calibration point 102 is the origin of the Cartesian grid. The worker then moves along the target wall TW, places the drill tip 82 at the corner between the target wall TW and the floor F, and signals the system 40 that the drill tip 82 is at a second calibration point 104. The system 40 combines the calibration points 102 and 104 to identify the axis (e.g., the X coordinate axis) of the Cartesian grid.

[0071] In one embodiment, the system 40 assumes by default that the target wall TW is vertical or nearly vertical. Alternatively, the worker may identify a third calibration point 106 in the upper corner between the target wall TW, the side wall W, and the ceiling C and / or a fourth calibration point 108 in the opposite corner between the target wall TW, the side wall W, and the ceiling C. Given the three calculation points 102, 104, and 106, the system 10 may calculate the orientation of the target wall TW, such as whether it is vertical.

[0072] After creating the artificial reference frame 100, the worker can use the system 40 to perform or identify work. The worker can identify a point 68 in the wall W where a hole needs to be drilled. The point 68 can be identified by positioning the drill tip 82 at the point 68 and switching a switch on the tool 72. The worker can store drill point information 90 about the point 68, such as the angle, diameter, and / or depth of the hole to be drilled at the point 68. The drill point information 90 can be provided to the system 40 using an interface 92. Optionally, the system 40 can store default drill point information 90 for the point 68 unless otherwise identified by the worker. After the point 68 (and optionally the drill point information 90) is identified, the system 10 can include an autonomous device (such as a drone) to perform the work identified by the worker at the point 68.

[0073] As the worker walks along the wall target TW where the artificial reference frame 100 is located, the system 40 calculates the position 52 of the drill tip 82 compared to the artificial reference frame 100 created by the worker. The artificial reference frame 100 may be, for example, Figure 5 A two-dimensional grid is shown, or it may be a three-dimensional grid.

[0074] Instead of using a level or tape measure to measure the piece, the worker can use an artificial reference system 100. The worker reads the location 52 of the drill tip 82 on the interface 92 of the tool 72. Once the drill tip 82 is at the location where the worker wants to mark a point 68, the worker marks the first point 68. The worker moves along the target wall TW to place another point 68. As the worker moves along the target wall TW, the interface 92 on the tool 72 optionally displays one or both of the location 52 of the drill tip 82 and the differential distance 94 between the drill tip 82 and the point 68.

[0075] Based on the information provided via the interface 92 of the tool 72, the worker can identify points 68 on the target wall TW that are perpendicular to each other, horizontal to each other, and / or equidistant from each other without the need for a measuring tape or a level. Among other advantages, the system 40 combines and even improves upon standard measuring tapes and levels to provide the worker with a quick and accurate measurement of the points 68.

[0076] Once the artificial reference frame 100 is established, the augmented image 78, such as displayed in the AR headset 46, can show artificial lines at the same distance from the ground. For example, a worker can establish a plane by identifying at least three calibration points 102, 104, and 106. The three points define the plane, and the intersection between the plane and the wall or object can be shown to the user in augmented reality, such as via the AR headset 46. In another example, the worker can establish the height above the floor F with only one calibration point 102, and a line indicating the height can be shown to the user in augmented reality, such as via the AR headset 46.

[0077] Another use of the artificial reference system 100 is to display offsets. For example, the worker identifies a point 68 along the wall W, selects a distance, and the headset can display the position(s) at that distance from the point 68 (e.g., the user locates a point and 3', and the headset shows the point(s) 68 3' from the point). This can also be used to provide offsets, such as the worker identifies a point 68 along the wall W, selects an offset distance, such as 3', and the worker also indicates how many additional points 68 to generate based on the offset. Thus, if the worker requests two additional points, the system 10 adds a second point 68 that is 3' offset from the first point 68, and the system also adds a third point 68 that is another 3' away from the second point 68 along the wall W and is therefore 6' away from the first point 68. These steps can be performed to create a virtual grid, a virtual routing, and can also be mirrored or extended to the ceiling and / or floor.

[0078] Go to Figure 6 , a virtual image 54 of the room and the worker can be streamed for remote viewing. For example, the location 52 of the AR headset 46 can be identified and then transmitted to a remote computer with a display, such as a second AR headset 46. The second AR headset 46 receives the location 52 and the construction plan 56 and generates an image 54 for the second worker to view. Figure 6 , the virtual image 54 is from the perspective of the virtual representation 130 showing the object 60 and the first worker. This dominant perspective allows the second worker to see not only what the first worker is seeing, but also what he / she may not be able to see near the first worker (e.g., if the first worker is close to the object). Alternatively, the virtual image 54 shown to the second worker may be the same as from the perspective of the first worker in the room R. The virtual image 54 may be generated based on the location 52 of any spatially aware tool 72. For example, the virtual image 54 may be based on the location 52 of the tool 72, the pole 122, or any other device 70.

[0079] Go to Figure 7, the tool 72 carried by the first worker (such as, the AR headset 46 worn by the first worker) can have a camera 98 that captures the actual image 96 that the first worker is seeing. The actual image 96 includes the wall W, the ceiling C, the floor F, and the object 62. The actual image 96 (such as, Figure 7 ) is streamed and displayed to the remote worker, such as via an AR headset 46 or a projector 44.

[0080] Go to Figure 8 , the augmented image 78, which is a combination of the virtual image 54 and the actual image 96, can be streamed to the remote worker. The streamed augmented image 78 allows the remote worker to see the actual image 96 augmented by the object 60 from the construction plan 56.

[0081] Go to Figures 9 to 11 , a worker may use system 40 to identify point 68. After point 68 is identified, the same worker may return and perform the identified work (e.g., drill out point 68), or another worker may synchronize their tool 72 or AR headset 46 with the artificial reference frame 100 including point 68 and then perform the identified work. Point 68 includes information 90 that describes characteristics of point 68, such as a desired hole depth and / or a desired hole width.

[0082] The worker may also identify objects 60, such as window objects 60 (in Fig.10 ). A worker may track the outline of the window object 60 via the tool 72, the pole 122, and / or any of the tools 72. After the window object 60 is identified and tracked by the worker, the window object 60 may subsequently be seen by the same worker or another worker.

[0083] Alternatively, the dimensions of the object 60 may be tracked and subsequently used to construct a physical representation of the object 60. For example, if a worker tracks the position and dimensions of a marble countertop, the tracked dimensions of the object 60 may be saved and provided to a party who cuts the marble countertop to the measured dimensions. The measured and transmitted dimensions may be in 2D, and a third dimension assumed or otherwise selected, or the measured and transmitted dimensions may be in 3D.

[0084] Go to Fig.11, the worker can also use a device 70 (such as an AR headset 46 or a pole 122) to identify objects 60 to be installed in and around the room R. The worker wearing the AR headset 46 views an augmented image 78 that includes a selection menu 132 that overlays the physical image 74 and the rest of the augmented image 78. The selection menu 132 includes a component 134 and a component name 136. In this example, the component 134 is an HVAC component. The worker can select the component 134 from the selection menu 132 and identify the location 52 in the room R where the component 134 will be placed. After the worker identifies the component 134 to be placed in the room R, the component 134 becomes a virtual object 60. The worker can continue to identify components 134 to add to the augmented image 78 in the room. As components 134 are added, the order list 138 is updated to reflect the quantity and identification of the components 134 that the worker has identified. After the worker has completed the virtual construction activities, the order list 138 can be submitted for ordering, such as directly to a supplier or to a manager (e.g., a general contractor) to be combined with other order lists 138. When performing construction on object 60, the worker can select different types of parts in addition to HVAC parts, such as ductwork parts, lighting, conduit (e.g., conduit elbows), rod frame rafters, ironwork, walls, sprinkler systems, data communication parts, and tile work.

[0085] In one embodiment, the device 70 continuously updates its location 52. In another embodiment, the device 70 updates its location 52 intermittently.

[0086] refer to Figures 12 to 22 , identifying the various tools used in system 40. Fig.12 , a projector 44 on a tripod 110 displays an image 54 with an object 60, in this case a sink object 60. The projector 44 has a height 112 that can be adjusted by adjusting the tripod 110. The position at which the projector 44 displays the image 54 can also be adjusted by rotating the projector 44 on an axis 114. The projector 44 includes a number of receivers 120 for calculating the position 52 of the projector 44. As the projector 44 moves around the room R, the projector 44 updates the image 54 based on the construction plan 56 or the artificial reference system 100 and the point 68.

[0087] exist Fig.13, the flashlight projector 116 displays an image 54. The flashlight projector 116 calculates its position 52 based on the calibration signal 150 detected by the receiver 120. Based on its position 52, the flashlight projector 116 generates an image 54 to be displayed. Based on the construction plan 56, the flashlight projector 116 enhances the image 54 to include the object 60. The flashlight projector 116 includes a number of receivers 120 for calculating the position 52 of the projector 44. As the flashlight projector 116 moves around the room R, the flashlight projector 116 updates the image 54. The flashlight projector 116 includes an interface 92. The interface 92 receives input (e.g., via an input device such as a keyboard) and provides output to the worker (e.g., via an output device such as a display).

[0088] exist Fig.14 , the light station 42 includes a transmitter 118. The transmitter 118 emits light (such as infrared light) to enable the tools 72 to calculate their position 52. Many different embodiments of the light station 42 include multiple transmitters 118. For example, the position 52, timing, and direction of light emissions from the light station 22 can be known by the receiver 120 and relied upon to calculate the relative position 50 of the receiver 120 when receiving from a single light station 42, or the position 52 of the receiver 120 when receiving light from multiple light stations 42.

[0089] exist Fig.15 , the sensor 48 includes a number of receivers 120. The receivers 120 detect light (such as infrared light emitted by the light station 42) to calculate the position 52 of the sensor 48. The sensor 48 can be used to calibrate the position of the tool 72, such as the position 52 of the drill tip 82.

[0090] exist Fig.16 , the AR headset 46 includes a receiver 120 and a camera 98. The receiver 120 interacts with a light emitter (such as a light stand 42) to determine the position 52 of the AR headset 46. The camera 98 is used to capture what the worker is actually looking at.

[0091] exist Fig.17, a pole 122 is used by a worker to work in a room R. The pole 122 includes a receiver 120 for calculating the position 52 of the pole 122. For example, the pole 122 includes a receiver 120 at a handle end 124 and a receiver 120 at a lower end 126. The pole 122 includes an interface 92 that provides output and receives input from a worker. The output to the worker may include informing the worker of the location to go, such as, via an indication on a display screen, a vibration pattern, a sound, and / or an overhead view image of the worker's location relative to their desired location. The pole 122 may also be used as a marking tool to indicate a point 68. The pole 122 may include an interface button 92 that, when pressed, digitally marks the point 68 or creates a digital line. The pole 122 may provide feedback to the worker by indicating where the tip of the pole 122 is located. The pole 122 may also display an arrow to the worker indicating the direction the worker should travel. The interface 92 on the pole 122 , which in some embodiments includes a display, may also display an augmented image 78 in which the point location 68 is imaged within the physical environment as seen by a camera of the device 70 .

[0092] Another way that the pole 122 can convey directions to the worker is to show a hollow circle on a display, where the display represents the target destination. As the worker walks around carrying the pole 122, the worker's representation moves in the display. The worker's goal is to align him / herself on the hollow circle.

[0093] exist Figure 18 to Figure 19 , the robotic total station 140 interacts with the light stand 42 to identify the location 52 of the robotic total station 140. The robotic total station 140 includes an interface 92, a camera 98, and a tripod 110. The robotic total station 140 can be used to identify a point 68, such as by a worker selecting a point 68 using a device 70, or by the robotic total station 140 projecting light to indicate the location where a previously selected point 68 is located. The robotic total station 140 can also calculate the distance between two locations (e.g., between a point 68 and the robotic total station 140, between a first point 68 and a second point 68). The robotic total station 140 can also perform a 3D scan of the room R where the robotic total station 140 is located, and perform a volume calculation of the room R based on the scan. The robotic total station 140 can also track a device 70 including a receiver 120 that includes a prism for reflecting and / or refracting light. The robotic total station 140 may also determine whether an object (such as a wall W) is plumb / square, such as by scanning the object.

[0094] exist Fig. 20, a tripod 110 with an articulating laser 142 is depicted. The light emitted by the articulating laser 142 is used to identify points 68, anchor points, risers, wall locations, etc. The light emitted by the articulating laser 142 can interact with the remote receiver 120 to determine the location 52 of other equipment 70. The transmitter 118 emits light at the reflector 162, and the reflector 162 adjusts its orientation to redirect the location to which the light from the transmitter 118 is reflected. The articulating laser 142 includes a rotating base with a worm and spur gears and an adjustable lead screw 166. The articulating laser 142 can determine its 3-D location 52 by emitting light in the form of a vertical line of laser light and scanning the room R to find active laser receivers 120 placed around the work site at known points 144. Once located, the articulating laser 142 can determine its distance to at least two of these beacons by triangulating its location 52. The articulating laser 142 may also determine its position 52 based on learning the position 52 of the tool 72 mounted to the articulating laser 142 .

[0095] exist Fig.21 , a laser receiver 146 with a vertical line laser is depicted. In one embodiment, the laser receiver 146 with a vertical line laser may be a QML800 commercially available from Spectra, or a similar laser receiving and / or transmitting device.

[0096] exist Fig. 22 , a multi-directional camera system 148 is depicted. The multi-directional camera system 148 includes several cameras 98.

[0097] Go to FIG. 23A to FIG. 23C , the robotic total station 140 receives the location 52 of the known point 144. The robotic total station 140 calculates its location 52 based on identifying the known point 144 in the room R. An articulating laser 142 is placed in the room R. The articulating laser 142 calibrates its location 52 by interacting with the robotic total station 140, and then displays the point 68 for the worker to perform the action. The articulating laser 142 displays the point 68 by projecting laser light on the relevant location 52 of the ceiling C, wall W, or floor F. The articulating laser 142 identifies the point 68 within the projection range 152 by referencing the construction plan 56 and / or the artificial reference system 100. The projection range 152 of the articulating laser 142 is the area on which the articulating laser 142 can project the point 68. A second articulating laser 142 can be added to the room R after the location 52 is calibrated. Fig.23C ) and display point 68 without reference to the robotic total station 140.

[0098] Go to FIG. 24A to FIG. 24D, a laser receiver 146 with a vertical line laser and a tripod 110 are placed in the room R. The tripod 110 may be coupled to an articulating laser 142, a projector 44, or other device 70 that projects light for workers in the room R. The tripod 110 is placed on the laser receiver 146 with a vertical line laser, and the tripod 110 calibrates its position 52 in the room R, such as by identifying the corners of the room R and calculating the position 52 of the tripod 110 based on the relative position of the corners of the room R ( Fig. 24B ). The tripod 110 has the same horizontal position 52 as the laser receiver 146 with vertical line laser, so the tripod 110 transmits its horizontal position 52 to the laser receiver 146 with vertical line laser. Then, the tripod 110 is moved and recalibrated with respect to the laser receiver 146 with vertical line laser ( Fig.24C ). The tripod 110 identifies a point 68 within a projection range 152 by referencing the construction plan 56 and / or the artificial reference system 100. The projection range 152 of the tripod 110 is the area on which the tripod 110 can project the point 68. A second tripod 110 may be added to the room R ( Fig.24D best shown) and point 68 is displayed.

[0099] Go to FIG. 25A to FIG. 25C , a plurality of sensors 48 are arranged around the room R. A tripod 110 with a light stand 42 is placed in the room R. A known point 144 at which the sensor 48 is located is loaded into the light stand 42. The light stand 42 calibrates its position 52 (eg, by emitting light in the room R) compared to the sensor 48 at the known point 144. Fig.25B ), the sensor 48 can sense the light and calculate the relative position 50. The position 52 of the light stand 42 is calculated based on the relative position 50 of the sensor 48. The light stand 42 then projects a point 68 ( Fig.25C ).

[0100] Go to FIG. 26A to FIG. 26C , the pole 122 calibrates its position 52 to the light table 42 in the room R. The pole 122 then identifies the point 68. After the worker moves within the room R, the pole 122 projects additional points 68 in the room R. The pole 122 knows the point 68 based on the construction plan 56 and / or the artificial reference system 100.

[0101] Go to FIG. 27A to FIG. 27C, the pole 122 with the multi-directional camera system 148 calibrates its position 52 based on detecting a calibration signal 150 from an emitter 118, such as an infrared emitter 118. The pole 122 with the multi-directional camera system 148 then projects points 68 in the room R. After the worker moves within the room R, the pole 122 with the multi-directional camera system 148 projects additional points 68 in the room R. The points 68 are known to the pole 122 with the multi-directional camera system 148 based on the construction plan 56 and / or the artificial reference system 100.

[0102] Go to FIG. 28A to FIG. 28E , the multi-directional camera system 148 projects a rapidly rotating (e.g., 50 Hz, 60 Hz) light projection 154 (at Fig.28B 100). As a result, the multi-directional camera system 148 functions similarly to the light table 22 in that its projection of light can be used by the receiver 120 to calculate their relative positions 50. In a number of different embodiments, the light projection 154 is rotated by the multi-directional camera system 148 via digital rather than mechanical means, thereby providing a more stable rotation. Based on the sensor 48 receiving the light projection 154, the multi-directional camera system 148 calibrates its position 52. The multi-directional camera system 148 projects a point 68 in the room R. After the worker moves within the room R, the multi-directional camera system 148 projects additional points 68 in the room R. The multi-directional camera system 148 knows the point 68 based on the construction plan 56 and / or the artificial reference system 100.

[0103] Go to FIG. 29A to FIG. 29B , the projector 44 on the tripod 110 calibrates its position 52 based on receiving the calibration signal 150 from the light stand 42. Based on the construction plan 56 and the position 52, the projector 44 displays the image 54 including the virtual object 60. The worker references the image 54 and marks points 68 on a surface such as a wall W or a floor F (for example, the worker may mark an X or a circle where a hole is to be drilled for the sink object 60). After completing the work using the image 54, the worker moves the projector 44 and the tripod 110 to a new location. The projector 44 recalibrates its position 52 and then projects the image 54. In this embodiment, the projector 44 does not continuously update its position 52, but only calibrates the position 52 when the worker signals a calibration position (such as when the worker clicks a calibration button on the interface 92).

[0104] Go to FIG. 30A to FIG. 30B, the flashlight projector 116 calibrates its position 52 based on receiving the calibration signal 150 from the light table 42. Based on the construction plan 56 and the position 52, the flashlight projector 116 displays an image 54 including a virtual object 60. The worker references the image 54 and marks points 68 on a surface such as a wall W or a floor F (e.g., the worker may mark an X or a circle where a hole is to be drilled for a sink object 60). After completing the work using the image 54, the worker moves the flashlight projector 116 to a new position. As the flashlight projector 116 is moved, it continuously updates its position 52 and correspondingly updates the image 54 to reflect its new position 52.

[0105] Go to Fig.31 , the room R includes several light tables 42 and known points 144. Spatial awareness tools 72 (such as AR headsets 46 and poles 122) calculate their respective positions 52 based on the light tables 42 and / or the known points 144. In one embodiment, the tools 72 and poles 122 calibrate their positions 52 based on receiving calibration signals 150 from the light tables 42, and the AR headset 46 calibrates its position 52 based on identifying the known points 144 and their respective positions 52. Multiple spatial awareness devices 70 (e.g., tools 72, poles 122, AR headsets 46) are used simultaneously in the room R.

[0106] Go to FIG. 32A to FIG. 32F , the lamp stand 42 jumps across the room R. Initially, the lamp stand 22 is placed at a known location 52 in the construction plan 56. The pole 122 calibrates its location 52 to the lamp stand 42 and indicates a point 68 on the floor F. The two lamp stands 42 move to the new location 156 ( Fig.32C ), and all the light stands 42 are calibrated to the pole 122. The pole 122 calculates its position 52 based on the two unmoved light stands 42, and calculates the position 52 of the light stand 42 at the new position 156 based on this information. The position 52 of the light stand 42 at the new position 156 is transmitted to the light stand 42 for future calibration. The pole 122 indicates the point 68 on the floor F. Then, the three light stands 42 are moved to the new position 156 ( Fig.32F ), the poles 122 calculate their positions based on the previously moved lighthouses 42, calculate the positions 52 of the lighthouses 42 at the new positions 156, and transmit their positions 52 to the lighthouses 42 at the new positions 156.

[0107] In this series of images, four light stands 42 jump around in room R, repositioning two or more light stands 42 in each movement step. In other embodiments, room R may have any number of light stands 42 (including only two light stands 42), and any subset of light stands 42 may be repositioned in a given movement step (including moving a single light stand 42).

[0108] Go to FIG. 33A to FIG. 33D , the lamp stand 42 on the tripod 110 jumps across the room R. Initially, the lamp stand 42 on the tripod 110 calibrates its position 52 relative to the known point 144. The pole 122 calibrates its position 52 relative to the lamp stand 42 ( Fig.33B ), and identify point 68. The second light stand 42 and the second tripod 110 are calibrated relative to the first light stand 42 on the first tripod 110 ( Fig.33C ). In short, the second light stand 42 and the second tripod 110 are indirectly calibrated relative to the known point 144 via the first light stand 42 on the first tripod 110 .

[0109] The first lamp stand 42 is then moved to a new position 156 and its new position 52 is calibrated relative to the second lamp stand 42 ( Fig.33D ). Light stand 42 may continue to jump across room R as needed. To reduce position errors compounded by subsequent jumps, light stand 22 may recalibrate position 52 relative to known point 144 if / when known point 144 is available for reference.

[0110] Go to Fig.34 In one embodiment, the AR headset 46 includes a plurality of stereo cameras 158 arranged to be separated from each other. Based on the marginal difference between the images received by each camera 98 in the stereo cameras 158, the position 52 is calculated. For example, the corner between the target wall TW, the wall W, and the ceiling C is at a specific position for each camera 98, and the position depends on the distance between the corner and each camera 98. Therefore, the images captured by each camera 98 can be analyzed to determine the position 52 of the AR headset 46 on which the stereo camera 158 is located.

[0111] Go to Fig.35 , physical objects 62 are not always installed in room R exactly where construction plans 56 indicate they should be installed. In one embodiment, image 54 projected by projector 44 is an as-built model 160. As-built model 160 reflects the locations where underlying physical objects 62 were actually installed or created when room R and associated objects 62 were constructed. As-built model 160 is generated while the building is being constructed, or it can be generated after the fact based on indicating to system 400 where objects 62 exist.

[0112] In a number of different embodiments, the present disclosure describes that the devices 70 locally calculate their positions 52 based on the time at which the calibration signal 150 is received by the receiver 120. In one embodiment, the device 70 measures the timing of receiving the calibration signal 150 and transmits the timing to a remote computer that performs the calculation. Subsequently, the results of the calculation (such as the position 52) are transmitted back to the device 70. In one embodiment, the timing of the detected calibration signal 150 is transmitted to the remote computer, which then calculates the position 52 and then streams the image 54 back to the device 70 and / or sends a command to the device (e.g., if the tool 72 is close to a gas pipeline, then deactivate the tool).

[0113] It should be understood that the drawings show exemplary embodiments in detail, and it should be understood that the present application is not limited to the details or methods set forth in the description or shown in the drawings. It should also be understood that the terminology is for descriptive purposes only and should not be considered as limiting.

[0114] In view of this description, other modifications and alternative embodiments of many different aspects of the present invention will be obvious to those skilled in the art. Therefore, this description is interpreted as being illustrative only. The construction and arrangement shown in a plurality of different exemplary embodiments are illustrative only. Although only a few embodiments are described in detail in the present disclosure, many modifications are possible (e.g., changing the size, dimensions, structure, shape and proportion of a plurality of different elements, the value of parameters, the installation arrangement, the use of materials, colors, orientations, etc.) without substantially departing from the novel teachings and advantages of the subject matter described herein. Some elements shown in the integral forming may be composed of a plurality of parts or elements, the position of the elements may be reversed or otherwise changed, and the nature or quantity or position of the discrete elements may be changed or changed. According to alternative embodiments, the order or sequence of any process, logic algorithm or method steps may be changed or reordered. Without departing from the scope of the present invention, other substitutions, modifications, changes and omissions may also be made in the design, operating conditions and arrangement of a plurality of different exemplary embodiments.

[0115] Unless otherwise stated, any method set forth herein should never be interpreted as requiring that its steps be performed in a particular order. Accordingly, in the absence of an actual listing of the order in which the steps of a method claim should be followed or in the absence of a specific statement in the claim or specification that the steps should be limited to a particular order, it is by no means intended that any particular order can be inferred. In addition, the article "a" used herein is intended to include one or more parts or elements and is not intended to be interpreted as only one. As used herein, "rigidly coupled" refers to two parts coupled in such a manner that when subjected to force, the parts move together in a fixed positional relationship.

[0116] A number of different embodiments of the present invention are directed to any combination of any features, and any such combination of features may be claimed in this or a future application. Any feature, element, or component of any exemplary embodiment discussed above may be used alone or in combination with any feature, element, or component of any other embodiment discussed above.

Claims

1. A method of displaying information from a construction plan, comprising: electronically tracking the position and orientation of an object; calculating a spatial relationship between the construction plan and the position and orientation of the object; generating augmented data based on a spatial relationship between the construction plan and the location and orientation of the object; calculating the relationship between the construction plan and the position and orientation of the tool; transmitting a signal to deactivate a function of the tool based on a spatial relationship between the construction plan and the position and orientation of the tool; displaying an image representing the augmented data; as well as receiving a command to switch between displaying a generic image of the construction object and displaying a schematic representation of the construction object, wherein the image representing the augmented data comprises a representative image of a construction object within the construction plan, and wherein the construction object is selected from the group consisting of a door, a window, an appliance, and an electrical socket, and Wherein, the object includes a spatial perception device.

2. The method of claim 1, wherein: The representative image is selected from the group consisting of a generic image of the construction object and a schematic representation of the construction object.

3. The method of claim 1, wherein: Displaying the image representing the augmented data includes projecting a representative image of the construction object at a physical location indicating an expected location of the construction object according to the construction plan.

4. The method of claim 1, further comprising: identifying a physical location corresponding to the landscape at which an image representing the augmented data is projected; as well as Calculating the relationship between the construction plan and these physical locations corresponding to the landscape; Therein, generating the enhanced data is further based on a relationship between the construction plan and the physical locations corresponding to the landscape.

5. The method of claim 1, further comprising: receiving information representing a position of a physical object; comparing information representing the position of the physical object to the position and orientation of the tool; as well as As a result of the comparison, a signal is emitted to deactivate the function of the tool.

6. The method of claim 1, wherein: The object is manipulated by a first user, and wherein augmented data is provided to a second user remotely located relative to the first user.

7. The method of claim 1, wherein: The object includes a handheld projector, and wherein displaying an image representing augmented data includes: the handheld projector projects the image representing augmented data, and wherein the handheld projector changes the image representing augmented data due to detecting a change in at least one of a position of the handheld projector and an orientation of the handheld projector.

8. A method of displaying information from a construction plan, comprising: tracking a position and orientation of a first object; tracking a position and orientation of a second object; calculating a position and orientation of a third object based on the position and orientation of the first object and the position and orientation of the second object; calculating a relationship between the construction plan and the position and orientation of a third object; generating augmented data based on a relationship between the construction plan and the position and orientation of the third object; calculating the relationship between the construction plan and the position and orientation of the tool; transmitting a signal to deactivate a function of the tool based on a spatial relationship between the construction plan and the position and orientation of the tool; as well as displaying an image representing the augmented data, The image includes a representative image of a construction object within the construction plan, wherein the construction object is selected from the group consisting of doors, windows, electrical appliances, and electrical sockets.

9. The method of claim 8, wherein: Displaying images includes: projecting enhanced images.

10. The method of claim 8, further comprising: calculating a position and orientation of a fifth object based on the position and orientation of the third object; calculating a second relationship between the construction plan and the position and orientation of the fifth object; generating augmentation data based on a second relationship between the construction plan and the position and orientation of the fifth object; as well as The enhanced data is displayed to the user.

11. The method of claim 8, wherein: The third object includes a projector that projects an image representing the augmented data, the method further comprising: calculating a position and orientation of a fifth object based on the position and orientation of the first object and the position and orientation of the second object; calculating a second relationship between the construction plan and the position and orientation of the fifth object; generating second augmentation data based on a second relationship between the construction plan and the position and orientation of the fifth object; and A second image representing the second enhancement data is displayed.

Images