Helmet with integrated electronic subsystem

Abstract

Various improvements to equipment used on a construction site are presented. The described improvements relate to one or more of helmets, handheld controllers, tracking beacons, and charging stations, as well as subassemblies and methods of use of these elements. The described improvements may be used to augment the display of augmented reality information on a construction site. A safety helmet having at least one integrated electronic subsystem including an inner portion and an outer portion is described. The integrated electronics subsystem may provide computations for augmented reality displays.

Description

Technical Field

[0001] Certain aspects of the present invention relate to devices used on construction sites. Specifically, some examples relate to a set of components for displaying augmented reality (AR) information on construction sites. This set of components includes a safety helmet, a removable portion of the helmet, a controller, a tracking beacon, and a battery charging station. Methods of use are also described. Certain aspects can be used for applications beyond augmented reality on construction sites. Background Technology

[0002] Construction sites present unique challenges for the design and use of construction tools. Historically, sophisticated electronic equipment has been rare on construction sites. The design phase of a building typically takes place away from the construction site. It can involve designers or architects generating a three-dimensional (3D) model called a Building Information Model (BIM), which represents the structure to be built. The design phase is usually performed in an office using high-specification computer workstations. After the design phase, the BIM model is used to create a set of two-dimensional (2D) drawings (e.g., “blueprints”) that are sent to the construction site. There, they are used to manage and guide construction. Paper is preferred because it is relatively durable and can be easily reprinted if details are lost or damaged.

[0003] In recent years, progress has been made in integrating the design and construction phases of building projects. For example, WO2019 / 048866 A1 (also disclosed as EP3679321), incorporated herein by reference, describes a head-mounted device for displaying virtual images of BIM within a construction site. In one example, the head-mounted device includes a head-mounted object with one or more position tracking sensors mounted thereon, augmented reality glasses containing at least one display, and an electronic control system. The electronic control system is configured to convert the BIM, defined in an external real-world coordinate system, into an intrinsic coordinate system defined by the position tracking system, receive display position data from a display position device and head-mounted tracking data from the head-mounted tracking system, and render a virtual image of the BIM relative to the position and orientation of the head-mounted object on the construction site, and transmit the rendered virtual image to a user-viewable display. This effectively allows the user to view the proposed construction represented in the BIM as a virtual overlay on top of a view of the construction site seen through the augmented reality glasses.

[0004] WO2019 / 048866A1 illustrates a safety helmet (e.g., as shown in Figure 12 of this publication) and a handheld controller (e.g., as shown in Figure 12 of this publication). Figure 6An earlier iteration of the helmet (shown). The helmet has a set of electronic components powered by a fixed internal rechargeable battery cell. The rechargeable battery cell can be recharged by coupling the battery cell to a power source via a power connector socket (see, for example, paragraph

[0256] of this publication). A set of augmented reality glasses is also provided, which is mounted within a set of safety goggles that form part of the helmet. The augmented reality glasses also have a fixed internal rechargeable battery cell, which is also connected to a power connector socket for charging the battery cell.

[0005] US 2016 / 292918 A1 (also incorporated herein by reference) further illustrates an alternative design for a helmet that houses a set of display units for viewing augmented reality images. In US 2016 / 292918 A1, the helmet is adapted to receive the display units (see, for example, the publication's...). Figure 4B The display unit is connected to the wearable computer (see, for example, Figure 2 of this publication). Preferably, in US 2016 / 292918 A1, the wearable computer is worn by the user as a backpack. The wearable computer is further connected to a replaceable battery that powers the entire system.

[0006] Prototype systems (such as those shown in WO2019 / 048866 A1 and US 2016 / 292918 A1) have revealed numerous problems in their practical use on construction sites. Construction sites are noisy, dusty, and potentially hazardous environments for the use of electronic devices, especially those traditionally intended for home or office applications. Safety is often paramount, but this must be balanced with the comfort of users who frequently spend long periods on the site. Augmented reality within construction sites also offers opportunities for new forms of user interaction (e.g., human-computer interaction) for displaying and querying BIM data.

[0007] EP3508087 A1 describes a ballistic helmet system, for example, for use by military personnel, comprising a base layer configured to hold an integrated circuit layer electrically coupled to one or more power supply devices. An outer layer is used to hold the circuit layer and integrated devices. In the primarily illustrated embodiment, the battery pack includes a base that removably receives a powered shoe on a rear helmet bracket. In an alternative embodiment, the battery compartment includes mounting rails for attachment to the rear of the helmet. The battery compartment can be secured in place via threaded fasteners. The battery compartment includes a housing for receiving a rechargeable battery, such as a 3-volt lithium (CR123) battery. In some embodiments, the circuitry within the battery compartment includes a switch for selectively electrically coupling one of a group of individual battery cells. This switch may be a rotary switch on a circuit board including a control lever. The user uses the control lever to switch between different batteries.

[0008] GB2608001A describes a safety helmet for industrial workers, comprising an array of light-emitting elements, a communication module configured to provide bidirectional remote communication with a remote central controller, and a local controller configured to receive instructions from the central controller via the communication module and responsively operate the light-emitting elements. The color, number, and / or intensity of the operating light-emitting elements can be controlled. The helmet may include sensors that receive instructions from the central controller and transmit data to the central controller, wherein the data can be used to control the light-emitting elements. The helmet may include output devices that operate via instructions from the central controller. The light-emitting elements may be located in a chamber between an inner helmet layer and an outer helmet layer, wherein the outer layer may include diffuser elements.

[0009] CN214179334 U describes a smart helmet for safe construction. This invention discloses a structure for a smart helmet for safe construction, comprising a helmet shell with a matching inner shell, and a liner within the inner shell. The liner is hinged at its end to a chin strap, which has a locking clip. An integrated circuit board and a battery are arranged between the helmet shell and the inner shell, with the integrated circuit board electrically connected to the battery. The inner shell houses two speakers. A power switch is located on the helmet shell and is electrically connected to the battery. A camera is located at the front end of the helmet shell. A memory card is mounted on the liner. The camera is electrically connected to the memory card via the integrated circuit board. A microswitch is located on one side of the helmet shell and is electrically connected to the camera. An earphone is mounted on the helmet shell and electrically connected to the integrated circuit board. A flashlight is hinged to the helmet shell.

[0010] CN110934370 A describes an intelligent helmet system with real-time video monitoring capabilities, which can provide data for subsequent accident review and analysis. The safety helmet includes an outer shell, a cushioning mechanism, and an inner liner. The cushioning mechanism incorporates a positioning module, human body status sensors, a miniature camera, a buzzer, and a communication module.

[0011] GB2603496 A1 describes a head-mounted device for construction site work. This head-mounted device includes a head-mounted article, sensor devices for multiple positioning systems, each positioning system having a corresponding coordinate system, a head-mounted display for displaying a virtual image of a building information model, and an electronic control system having at least one processor. The at least one processor is configured to obtain a set of transformations mapping between the coordinate systems of the multiple positioning systems. This publication describes how to align multiple coordinate systems for rendering the information model.

[0012] CN107951114A describes a construction protective helmet. The helmet includes an outer protective layer. A fan is connected to the top of the outer protective layer, and an air intake is connected to the middle of the top of the outer protective layer.

[0013] US6122773A describes a ventilated safety helmet with an integrated fan.

[0014] US2013 / 254978 A1 describes a protective helmet and insert for reducing the likelihood or severity of a concussion. The protective helmet can be used for sports. The insert includes a shock-absorbing portion and a flexible liner portion, the shock-absorbing portion being disposed between the helmet shell and the liner portion. The shock-absorbing portion may have constant resistance to deformation forces to reduce the peak G-force applied to the head during an impact.

[0015] US 3758889A describes a shock-absorbing safety helmet or protective helmet of the helmet type, the helmet having a head-fitting suspension system removably interconnected within the helmet, the suspension system including freely crisscrossing crown straps, removable and size-adjustable headbands and neck straps, and removable soft and elastic sweatbands, the entire suspension system being mounted via suspension lugs at the free ends of the crisscrossing straps, the lugs having lateral shear pins and slidably suspended in retainers inside the helmet shell, the lugs and shear pins being designed to resiliently resist the lugs from engaging in the retainers, thereby increasing the absorption of impact impacts on the helmet. Summary of the Invention

[0016] The aspects of the invention are set forth in the appended independent claims. Variations of these aspects are set forth in the appended dependent claims. Unclaimed examples are also set forth in the following description. Attached Figure Description

[0017] Examples of the invention will now be described by way of example only with reference to the accompanying drawings, in which:

[0018] Figure 1A This is a schematic diagram of an exemplary enhancement system used on a construction site.

[0019] Figure 1B This is a diagram illustrating how BIM data is aligned with a view of the construction site.

[0020] Figures 2A to 2H These are schematic diagrams showing different views and configurations of an exemplary safety helmet with at least one integrated electronic subsystem.

[0021] Figures 3A to 3D This is a schematic diagram showing the internal and external portions of an exemplary safety helmet having at least one integrated electronic subsystem.

[0022] Figures 4A to 4E This is a schematic diagram illustrating an exemplary deformable ventilated connection for the inner and outer portions of an exemplary safety helmet.

[0023] Figures 5A to 5G This is a schematic diagram illustrating an exemplary battery connection interface and an exemplary removable housing portion.

[0024] Figure 6 This is a schematic diagram illustrating an exemplary location of the center of gravity of an exemplary safety helmet.

[0025] Figures 7A to 7C This is a schematic diagram illustrating an exemplary bracket height adjustment mechanism.

[0026] Figures 8A to 8D These are schematic diagrams showing different views of an exemplary handheld controller.

[0027] Figures 9A to 9C This is a schematic diagram showing different views of an exemplary tracking beacon.

[0028] Figure 10A and Figure 10B These are schematic diagrams showing different views of an exemplary battery charging station.

[0029] Figure 11 This is a schematic diagram illustrating the use of an exemplary handheld controller.

[0030] Figure 12A and Figure 12B This is a flowchart illustrating an exemplary method of manipulating virtual objects in an AR view using an exemplary handheld controller.

[0031] Figure 13 It is shown Figure 12B A schematic diagram illustrating the performance of an exemplary method.

[0032] Figure 14 This is a flowchart illustrating an exemplary method for preparing BIM data for augmented reality applications.

[0033] Figure 15A and Figure 15B It is shown that... Figure 14 A schematic diagram of an exemplary user interface associated with an exemplary method.

[0034] Figures 16A to 16L This is a schematic diagram illustrating a stage in an exemplary method for aligning a building information model with an augmented reality view.

[0035] Figure 17 This is a flowchart illustrating an exemplary method for aligning a building information model with an augmented reality view. Detailed Implementation

[0036] introduce

[0037] This specification proposes various improvements to equipment used on construction sites. The described improvements relate to one or more of the following: safety helmets, handheld controllers, tracking beacons, and charging stations, as well as sub-components and methods of use of these elements. The described improvements are particularly well-suited for enhancing the display of augmented reality information on construction sites. For example, the described aspects improve the comfort and ease of use for users, such as those wearing safety helmets with augmented reality displays.

[0038] This specification describes a number of different innovations. These innovations include, in particular: helmets with at least one integrated electronic subsystem comprising internal and external portions, such as a double-shell or double-layer design; deformable ventilated connectors for multi-layer helmets, providing airflow and energy absorption characteristics for the user's head; helmets with at least one integrated electronic subsystem including multiple battery connection interfaces for coupling multiple removable batteries, wherein these batteries may be "hot-swappable" while maintaining power to the subsystem; removable battery housings for the removable batteries in the helmet; kits for use on construction sites to provide augmented reality views of the construction site; helmet bracket height adjustment mechanisms; mobile controllers with distance measuring devices, such as handheld controllers; methods for using mobile controllers that allow users to interact with both the real and virtual worlds; and methods for preparing 3D BIM data for augmented reality applications. Each of these aspects can be applied and used individually or in any combination. Various details and advantages of each aspect are set forth with the corresponding description of that aspect.

[0039] The aspects described here improve upon the head-mounted devices described in WO2019 / 048866 A1 and / or US 2016 / 292918 A1. They offer greater user comfort, easier interaction with the virtual world, enhanced security, and a longer lifespan, among other benefits.

[0040] Definition of certain terms

[0041] Where applicable, the terminology used herein will be defined in accordance with the art. To facilitate the explanation of the following embodiments, explanations and definitions of certain specific terms are provided below.

[0042] The term "position tracking system" is used to refer to a component system for determining one or more of the position and orientation of an object within an environment. In some cases, the object includes a helmet or a handheld controller. The terms "positioning system" and "tracking system" can be considered alternative terms for "position tracking system," where the term "tracking" refers to repeatedly or iteratively determining one or more of the position and orientation over time. A position tracking system can be implemented using a single set of electronic components positioned on the object to be tracked, for example, a standalone system mounted in a head-mounted device. In other cases, a single set of electronic components located outside the object can be used. In some cases, a position tracking system may include a distributed system in which a first set of electronic components is positioned on the object to be tracked, and a second set of electronic components is positioned outside the object (e.g., as discussed later). Figure 1A and Figure 1B (As described). Electronic components may include sensors and / or processing resources (such as cloud computing resources). The location tracking system may include processing resources that can be implemented using one or more of embedded processing devices (e.g., on or within an object) and external processing devices (e.g., server computing devices). In a preferred example, the tracking system uses a kit of components that can be carried to the construction site (e.g., no remote server is required for use). References to data received, processed, and / or output by the location tracking system may include references to data received, processed, and / or output by one or more components of the location tracking system, which may not include all components of the location tracking system. Some location tracking systems described herein include externally mounted tracking beacons and devices such as safety helmets and handheld controllers with corresponding sensors. However, it should be noted that the different improvements described herein are not necessarily limited to the use of such location tracking systems, and said improvements can be used with other types of location tracking systems (e.g., systems based on stand-alone cameras).

[0043] The term "pose" is used herein to refer to the position and orientation of an object. For example, a pose may include coordinates and a set of angles, the coordinates specifying a position relative to a coordinate system, and the set of angles representing the orientation of a point or plane associated with the object within that coordinate system. For example, the point or plane may be aligned with a defining face of the object or a specific (reference) position on the object. In some cases, orientation may be specified as a normal vector or a set of angles relative to a defined orthogonal axis. In other cases, pose may be defined by multiple coordinates specifying corresponding multiple positions relative to the coordinate system, thereby allowing the orientation of a rigid body containing these points to be determined. For a rigid object, the position may be defined relative to a specific point on the object. A pose may specify the position and orientation of an object with respect to one or more degrees of freedom within a coordinate system. For example, an object may include a rigid body with three or six degrees of freedom. Three degrees of freedom may be defined as translation relative to each axis in 3D space, while six degrees of freedom may include a rotational component relative to each axis. In other cases, three degrees of freedom may represent two orthogonal coordinates in a plane and a rotational angle (e.g., [x, y, θ]). The six degrees of freedom can be defined by a vector of [x, y, z, roll, pitch, yaw], where the variables x, y, and z represent coordinates in a 3D coordinate system, and rotation is defined using a right-handed convention about the three axes, which can be the x, y, and z axes. In the examples involving head-mounted devices in this paper, pose can include the position and orientation of a defined point on the head-mounted device or on a head-mounted item forming part of the head-mounted device, such as the center point within the head-mounted item calibrated based on sensors positioned on the head-mounted item. In some cases, the pose of an object defined by the centroid of a reference object can be transformed into a pose defined at another point with a fixed relationship to the centroid; for example, the pose of a helmet defined relative to the center point within the helmet can be mapped to a pose indicating the position and viewing direction of a set of connected augmented reality glasses. It should be noted that the same position and orientation information can be represented using different coordinate systems (e.g., using different basis functions as axes), where the defined transformations can be converted between different coordinate systems. For example, a polar coordinate system can be used instead of a Cartesian coordinate system. In some cases, one or more of the following can be used to define the pose: a set of three Cartesian coordinates and a set of three Euler angles; a set of three Cartesian coordinates and a rotation matrix (e.g., mapping a set of axes of the object defined by the origin of the reference object to a set of axes of the reference coordinate system); a set of three Cartesian coordinates and a set of quaternions; and a homogeneous transformation matrix (e.g., mapping the origin of the object to the origin of the reference coordinate system).

[0044] The term "coordinate system" is used herein to refer to a reference system, such as that used by position tracking systems and BIM. For example, the pose of an object can be defined within a three-dimensional geometric space, where these three dimensions have corresponding orthogonal axes (typically x, y, z) within the geometric space. An origin (typically set to zero (0, 0, 0)) can be defined for a coordinate system to which the lines defining the axes intersect. The position of a coordinate system can be defined as points within the geometric space that reference unit measurements along each axis, such as x, y, and z values ​​representing distances along each axis. In some cases, quaternions can be used at least to represent the orientation of objects (such as head-mounted devices or cameras) within a coordinate system. In some cases, dual quaternions allow for the representation of both position and rotation. Dual quaternions can have eight dimensions (i.e., include arrays with eight elements), while normal quaternions can have four dimensions.

[0045] The terms “intrinsic” and “external” are used in some examples to refer to the coordinate system within a location tracking system and a coordinate system outside of either location tracking system, respectively. For example, an external coordinate system could be a 3D coordinate system used to define an information model (such as BIM) that is not directly associated with any positioning system, while an intrinsic coordinate system could be a separate system used to define points and geometry relative to sensor devices used for a particular location tracking system.

[0046] Some of the examples described herein use one or more transformations to convert between coordinate systems. The term "transformation" is used to refer to a mathematical operation that can be performed on one or more points (or other geometric structures) in a first coordinate system to map those points to corresponding positions in a second coordinate system or between points in the first coordinate system. For example, a transformation can map the origin, defined in the first coordinate system, to a point that is not the origin in the second coordinate system. Transformations can be performed using matrix multiplication. In some examples, a transformation can be defined as a multidimensional array (e.g., a matrix) with rotation and translation terms. For example, a transformation can be defined as a 4x4 (element-wise) matrix representing the relative rotation and translation between the origins of two coordinate systems. The terms "mapping," "transformation," and "machining" are used interchangeably to refer to using transformations to determine the position and orientation of an object originally defined in the first coordinate system relative to a second coordinate system. It can also be noted that the inverse of the transformation matrix mapping from the second coordinate system to the first coordinate system can be defined.

[0047] Some of the examples described herein refer to “parts” or “components” of an artifact. These may include removable or separable portions of the artifact that are fastened or otherwise engaged to produce an finished product. Components described as removable or separable may be removable or separable under specific circumstances, such as when assembled during manufacturing or disassembled during repair, and / or may be removable or separable in use, such as when performing one or more actions (such as decoupling or releasing a component).

[0048] Some of the examples described herein refer to a “safety helmet.” This is used to refer to a form of helmet worn on a user’s head to provide protection against one or more of falling objects, impacts, and electric shocks. The examples of safety helmets described herein have an external rigid portion that provides at least one protective element for the user’s head.

[0049] Some examples describe safety helmets used as “head-mounted devices.” The term “head-mounted device” is used to refer to a device suitable for use with or related to a person’s head, such as one mounted on or associated with the head. The term has a similar definition to its use in so-called virtual or augmented reality head-mounted devices. In some examples, head-mounted devices include head-mounted items (such as safety helmets), although head-mounted devices can be supplied as kits of detachable parts. These detachable parts can be removable and can be selectively assembled for use, but can be removed for repair, replacement, and / or when not in use.

[0050] Although the term “augmented reality” (AR) is used herein, it should be noted that this is considered to include so-called “virtual reality” (VR) methods, such as all methods, regardless of the level of transparency of the external view of the world. For example, the phrase “through-view” is sometimes used in the context of “virtual reality” to refer to an AR-like display of digital information on an image of the external world acquired by a camera on a VR headset. The use of the terms “augmented reality headset” or “augmented reality” covers such VR headsets used in through-view mode to provide AR information. The term “augmented reality” also covers so-called “mixed reality” (MR) methods, in which aspects of the virtual world are “mixed” with aspects of the real world. It should be noted that different terms are used depending on the manner to refer to similar methods of rendering a virtual representation of the view or capture aspect of the visible world. For ease of reference, all such methods are considered to fall under the currently used term “augmented reality,” in which a view of reality (e.g., the external world) is enhanced with rendered objects that do not exist in that reality.

[0051] Some of the location tracking systems described herein use one or more sensor devices to track an object. Sensor devices may include monocular cameras, stereo cameras, color cameras, grayscale cameras, event cameras, time-of-flight cameras, depth cameras, infrared cameras, active markers, passive markers, photodiodes for detecting electromagnetic radiation, radio frequency identifiers, radio receivers, radio transmitters, and light emitters including laser emitters. A location tracking system may include one or more sensor devices on the object. Some, but not all, location tracking systems may include external sensor devices, such as scanning beam tracking beacons or camera devices. For example, an optical positioning system for tracking an object with active or passive markers within the tracked volume may include an externally mounted grayscale camera plus one or more active or passive markers on the object. Some location tracking systems may use a combination of sensor devices to track an object, such as photoelectric sensors and camera components. In other examples, multiple location tracking systems using different sensor devices may be used, and sensor data from those tracking systems may be fused to display an augmented reality view.

[0052] Some examples provide head-mounted devices used on construction sites. The term "construction site" will be interpreted broadly and is intended to refer to any geographical location where an object is built or constructed. A "construction site" is a specific form of "environment," the real-world location where an object is situated. An environment (including construction sites) can be external (outer) and internal (inner). An environment (including construction sites) does not need to be continuous, but can include multiple discrete sites where objects can move between them. Environments include terrestrial and non-terrestrial environments (e.g., at sea, in the air, or in space).

[0053] The term “rendering” has a conventional meaning in the fields of image processing and augmented reality, and is used herein to refer to the preparation of image data that allows it to be displayed to a user. In this example, image data may be rendered on a head-mounted display for viewing. The terms “virtual image” or “augmented reality image” in the context of augmented reality refer to an image that can be overlaid on a view of the real world, such as an image overlay that can be displayed on a transparent or semi-transparent display when viewing a real-world object, or may include an image composed of a captured view of the line of sight and digital information. In some examples, a virtual image may include an image associated with an “information model.” The term “information model” is used to refer to data defined relative to an external coordinate system, such as information about the relative positioning and orientation of points and other geometric structures on one or more objects. For example, an information model may be defined relative to geodetic or geocentric coordinates on the Earth’s surface plus height (e.g., above a defined sea level or reference point). In the example described herein, data from the information model is mapped to known points in the real world tracked using one or more position tracking systems, such that the data from the information model can be appropriately prepared for display in the real world as a reference for tracking. For example, general information about the configuration of an object and / or the relative positioning of an object with respect to other objects, defined in a universal 3D coordinate system, can be mapped to a real-world view and one or more points in that view.

[0054] The terms “control system” and “electronic subsystem” are used herein to refer to a hardware architecture or combination of general-purpose hardware and specific software (e.g., specific computer program code executing on one or more general-purpose processors) having a specific function (e.g., mapping input data to output data). An “engine” or “control system” as described herein may be implemented as a chipset in a specific package, such as an application-specific integrated circuit (ASIC) or a programmable field-programmable gate array (FPGA), and / or software objects, classes, class instances, scripts, code segments, etc., as executed by a processor in use. The term “integrated electronic subsystem” is used herein to describe a physical component that operates by controlling the behavior of electrons within a material. The term “integrated” is used to indicate the fact that a subsystem is provided as part of a larger system (e.g., a safety helmet). A subsystem may be removable or fixed and typically has a defined location within the larger system for use. A subsystem may be installed within a larger system. The term “subsystem” is used to indicate the fact that certain electronic components provide one or more functions within a larger system. In one case, an integrated electronic subsystem includes a processor and memory to perform computations; for example, an integrated electronic subsystem may include an embedded computer (also called a computing module). Integrated electronic subsystems may also include sensors and / or sensor processing circuitry and / or components.

[0055] The term "camera" is broadly used to cover any camera device having one or more channels configured to capture one or more images. In this context, a camera can include a camera that outputs a series of images as image data over time, such as a series of frames constituting a "video" signal. It should be noted that any still camera capable of outputting a continuous series of images over time can also be used to implement camera functionality. References to cameras can include any light-based sensing technology, including event cameras and LIDAR sensors (i.e., laser-based distance sensors). Event cameras are referred to in the art as imaging sensors that respond to local brightness changes, where pixels can asynchronously report brightness changes as they occur, thus mimicking characteristics more like human vision.

[0056] The term "image" is used to refer to any array structure that includes data derived from a camera. Images typically comprise a two-dimensional array structure, where each element in the array represents the intensity or amplitude in a particular sensor channel. Images can be grayscale or color. In the latter case, the two-dimensional array can have multiple (e.g., three) color channels. Due to the lower dimension of grayscale images, they may be preferred for processing. For example, an image processed in the method described later may include the luminance channel of a YUV camera.

[0057] The term "connection" is used to refer to one or more components that allow physical and electronic communication, where the meaning is generally self-evident from the context of use. A physical connection can physically link two different workpieces. An electronic connection can allow analog and / or digital communication between two electronic devices. Additionally or alternatively, the electronic device can provide power. The term "interface" is similarly used to refer to one or more mechanical, hardware, and software interfaces used to connect two or more components. Hardware and / or mechanical interfaces can include complementary surfaces that allow rigid fit and, in some cases, allow the flow of electrical signals.

[0058] It should be noted that in this field, the acronym “BIM” is used to refer to both “Building Information Modeling” and “Building Information Model” (as the terminology has evolved naturally over the past 2-30 years). References to “BIM” or “the BIM” herein refer to a building information model, i.e., a building information modeling model. The term “BIM model” is sometimes used and is synonymous with the use of “BIM” or “the BIM”, both referring to a three-dimensional model of a building. The term “BIM data” is used to refer to data that defines at least a portion of a BIM model. References to BIM and BIM models also include references to portions of such models; for example, a complete model of a building may have thousands or hundreds of thousands of three-dimensional elements representing multiple different phases of construction across multiple different locations, and therefore only a subset of the complete model may be loaded at any given time.

[0059] Typical examples of AR displays on construction sites

[0060] Figure 1A The first example shown illustrates how augmented reality information can be displayed on a construction site. It should be noted that the location tracking system described in this example is provided for ease of understanding of the invention. While preferred, it should not be considered limiting; the invention can be applied to many different types of location tracking systems.

[0061] Figure 1A Location 1 in the construction site is shown. Figure 1AA position tracking system 100 positioned at location 1 is shown. In this example, the position tracking system 100 includes a laser-based position tracking system similar to that described in WO2019 / 048866 A1; however, this position tracking system is for ease of interpretation, and this embodiment is not limited to this type of position tracking system. In other embodiments, different position tracking systems may be used, including high-precision positioning systems based on optical markers, such as those provided by NaturalPoint, Inc. of Corvallis in Oregon, USA (e.g., their OptiTrack system), and monocular, depth, and / or stereo camera simultaneous localization and mapping (SLAM) systems. SLAM systems can be sparse or dense and can be feature-based and / or use trained deep neural networks. So-called direct systems can be used to track pixel intensity, and so-called indirect systems can be feature-based. Deep neural networks can be used to train indirect methods. Examples of “traditional” or non-neural SLAM methods include ORB-SLAM and LSD-SLAM, as described in Mur-Artal et al.’s paper “ORB-SLAM: a Versatile and Accurate Monocular SLAM System” published in IEEE Transactions on Robotics in 2015 and Engel et al.’s paper “LSD-SLAM: Large-Scale Direct Monocular SLAM” presented at the European Conference on Computer Vision (ECCV) in 2014, both of which are incorporated herein by reference. Example SLAM systems incorporating neural network architectures include Bloesch et al.’s “CodeSLAM – Learning a Compact Optimisable Representation for DenseVisual SLAM” (published at CVPR-2018 in conjunction with the Conference on Computer Vision and Pattern Recognition) and Tateno et al.’s “CNN-SLAM: Real-time dense Monocular SLAM with Learned Depth Prediction” (published at CVPR 2017), which are also incorporated herein by reference.It should also be noted that position tracking systems can also be based on neural network representations of 3D space, such as those based on neural radiance fields (“NeRF”), as described in the paper “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” by Ben Mildenhall et al., published on arXiv on March 19, 2020. Data from different methods can also be fused in a combined or “fused” system. For example, short-term (e.g., milliseconds or seconds) tracking can be performed in combination with one or more inertial measurement units (IMUs) installed within the tracked object.

[0062] exist Figure 1A In this example, the exemplary position tracking system 100 includes a plurality of spaced-apart tracking beacons 102. In one particular implementation example, the tracking beacons 102 include a device that can be selectively operated to emit omnidirectional synchrotron pulses 103 of infrared light, and include one or more rotors arranged, for example, to sweep across one or more linear invisible optical fan-shaped beams 104, 105 at position 1 on mutually orthogonal axes as shown below. Figures 9A to 9C Describe an example tracking beacon. In this example, tracking beacons 102 are spaced up to approximately 5-10 meters apart. Figure 1A In the example, four tracking beacons 102 are used, but in other embodiments, fewer than four tracking beacons 102 may be used, for example, one, two, or three tracking beacons 102, or more than four tracking beacons. It should be understood that for some forms of SLAM position tracking systems, the tracking beacons 102 may be omitted. As described in WO2019 / 048866A1, by sweeping laser beams 104, 105 across the construction site 1 at a precise constant angular velocity and synchronizing the laser beams 104, 105 with a precisely timed synchronization pulse 103, each tracking beacon 102 in the laser position tracking system can generate two mutually orthogonal spatially modulated beams 104, 105 in a time-varying manner. These two mutually orthogonal spatially modulated beams 104, 105 can be detected by photoelectric sensors within the tracked volume to locate the position and / or orientation of one or more tracked objects within the tracked volume. Other position tracking systems can use different techniques to track objects, including detecting one or more active or passive markers on the object as observed by a tracking device in the form of one or more cameras equipped with tracking beacons 102, and observing the volume being tracked. In SLAM systems, tracking can be performed based on data streams from one or more camera devices (and possibly additional odometry or inertial measurement unit (IMU) data).

[0063] Figure 1ATwo users 2a and 2b are also shown. Each user wears a helmet 10 with an integrated augmented reality head-mounted device, wherein the device has sensors arranged to detect signals emitted from one or more tracking beacons 102. The helmet 10 is configured to be located within position 1. Users 2a and 2b use the augmented reality head-mounted device to view virtual images of one or more interior partitions 52 and 58 via a head-mounted display (HMD), which are defined in the BIM and can be aligned with portions of the partial structure of the building 60.

[0064] As another example, Figure 1B A 3D BIM 110 of a building 50 to be constructed is shown. Building 50 has exterior walls 51, 52, 53, 54, a roof 55, and interior partitions, one of which is shown at 58. One of the walls 52 is designed to include a window 61. BIM 110 is defined relative to an external coordinate system, which may be a geographic coordinate system (e.g., a set of land coordinates) or a specific computer-aided design (CAD) reference origin. By configuring the alignment of BIM 110 with a first location 1, user 2a or 2b can see how a portion of the building under construction (such as window 61) matches the original 3D specifications of the building within the BIM. Adjustments can then be made to the building under construction if it is not being built according to specifications. The BIM may include multiple layers showing different parts of the building, such as services (electricity, gas, and / or communication piping), interior construction sections, and / or interior finishing. See later. Figure 15A and Figure 15B Provide further information about BIM components.

[0065] Exemplary safety helmet with integrated electronic subsystem

[0066] Figures 2A to 2H Different views and configurations of an exemplary helmet with an integrated electronic subsystem are shown. In this example, the integrated electronic subsystem is an augmented reality system that includes electronics for rendering augmented reality images within a set of augmented reality glasses. The integrated electronic subsystem may include a processor and memory (i.e., a computing architecture), but may exclude the set of augmented reality glasses and display circuitry (such as display drivers) for the glasses.

[0067] Figure 2A A front view 200 of a helmet 210 in a configuration without a removable sun visor is shown. Figure 2B A front view 201 of a safety helmet 210 with a removable sun visor 252 is shown. Figure 2AInitially, the helmet 210 includes an integrated set of safety goggles 220, a camera assembly 230, and left and right wing portions 240 forming part of a corresponding removable battery housing. Although the term "safety goggles" is used herein, it should be noted that "safety glasses" or "safety face shield" are fairly comparable synonyms. The integrated safety goggles 220 may be formed from a protective polymer designed to withstand impact and constructed according to defined optical wearing safety standards. The surface of the helmet 210 includes multiple photoelectric sensors 212 (ten are present in the view, but only two are labeled for clarity). These photoelectric sensors 212 may include photodiodes (such as silicon photodiodes) for detecting... Figure 1A and Figure 1B The tracking beacon 102 shown emits an electromagnetic signal. In one case, the photoelectric sensor 212 may include part of an optical position tracking system as described in WO 2019 / 048866 A1 or WO 2016 / 077401 A1, the latter also incorporated herein by reference. The optical positioning system may be inside-out (i.e., using a sensor on an outward-sensing device) or outside-in (i.e., positioning an object within a tracked volume generated at least partially by an external device such as the tracking beacon 102). In a preferred example, the photoelectric sensor 212 is mounted behind a hole in the surface of the helmet 210. The photoelectric sensor 212 may be sealed (e.g., with a silicone sealant, etc.) to prevent water ingress. In some cases, the photoelectric sensor 212 may be molded as part of the housing material to create a completely sealed housing. In other examples, the photoelectric sensor 212 may be attached to the surface of the helmet 210.

[0068] Safety goggles 220 form the outer protective boundary of the eyes. Behind safety goggles 220 is an augmented reality display (see example...). Figure 2DIn this example, the augmented reality display includes a transparent display, where images can be viewed when a user observes the external world. Therefore, when a user wears a helmet and looks out through the augmented reality display and safety goggles, they are able to view a virtual image that can be overlaid on the view of the external world. In other examples, the safety goggles 220 can be replaced by a closed-back VR or MR headset, in which a virtual reality view is layered over the view of the external world captured by camera assembly 230 and displayed on a non-transparent screen, such as a liquid crystal display (LCD) or a light-emitting diode (LED) display (including organic LED-OLED displays). Display technologies may additionally or alternatively include liquid crystal on silicon (LCOS) displays (including transparent LCOS displays), digital light processing (DLP) displays, and micro-LED displays, etc. The display technologies mentioned herein can be applied in both opaque and transparent forms (e.g., for VR or AR).

[0069] Camera assembly 230 includes one or more cameras that can be used to enhance the display of augmented reality information. In this example, four cameras are provided. From the wearer's perspective, these cameras are: a right grayscale wide-angle camera 232-A (e.g., with a field of view greater than 90 degrees, and including fisheye cameras with a field of view up to 180 degrees); a color camera 234, such as a red-green-blue (RGB) camera; a distance imaging camera 236, such as a time-of-flight camera and / or an infrared (IR) camera, for determining distance measurements based on the time it takes for a light pulse (such as a laser or infrared pulse) to travel to and reflect from the target; and a left grayscale wide-angle camera 232-B. The cameras provided as part of camera assembly 230 can be used for SLAM localization and tracking as described above and / or for tracking objects in the field of view. In a preferred embodiment, the cameras provide data for use in conjunction with a position tracking system implemented by tracking beacon 102 and photoelectric sensor 212. The camera can also be used to capture images and / or videos of portions of the construction site for reporting and work inspections. In some cases, virtual objects (e.g., 3D holograms) are overlaid in the image or video field of view. In some other examples, camera assembly 230 may also include an inertial measurement unit (IMU); in a preferred example, multiple IMUs are housed within a safety helmet but not within camera assembly 230. Camera assembly 230 may be coupled to one or more integrated electronic subsystems and intermediate circuitry.

[0070] Figure 2BThis illustration shows how a removable visor 252 can be snapped into place in front of the safety goggles 220. The removable visor 252 may include a polarizing filter that improves the visibility of virtual images in an augmented reality display under bright light conditions, such as outdoors on a sunny day. The removable visor 252 may include a "press-fit" design, which allows it to be removed by loosening the visor from the goggles 220 and surrounding frame portions. For certain specifications of the helmet 210, the removable visor 252 may be omitted.

[0071] Figure 2C A side view 202 of a helmet 210 without the removable sun visor 252 is shown. The side view is from the right side of the helmet 210 as seen from the front, which corresponds to the left side of the helmet 210 from the perspective of the user wearing the helmet 210. In this description, the elements of the helmet will be referred to as views taken from the wearer's perspective, because although the perspectives differ in the accompanying drawings, this view is consistent. Figure 2C In the image, one can see how the brim 218 of the helmet 210 extends upward at an angle towards the upper rear of the helmet 210, surrounding the camera assembly 230, safety goggles 220, and left wing portion 240-B. An integrated electronic subsystem 260 is mounted at the rear of the helmet 210. In this example, the integrated electronic subsystem 260 is mounted behind a rear housing 262, which includes a ventilation grille 264 to allow airflow through the subsystem for cooling. The integrated electronic subsystem 260 in this example includes an embedded computer, comprising at least one processor and memory (e.g., one or more of volatile and non-volatile memory). The rear housing 262 also includes an additional photoelectric sensor 212. It can be seen how the lower edge of the goggles 220 and the wing portion 240-B form a continuously upward-sloping line towards the rear of the helmet 210. The wing portion 240-B provides structural rigidity to the front of the helmet 210, where the safety goggles 220 and the augmented reality display (shown later) are integrated together. The wing 240-B can also be configured in a removable form to form the battery housing (see later). Figures 5A to 5G (As described). Although only a side view is shown, it will be apparent from the other views provided that the design is symmetrical and that the right side of the helmet 210 will have a similar corresponding feature.

[0072] exist Figure 2C In this example, multiple ventilation holes 214 are visible in the top of the helmet 210. There are three ventilation holes 214 on each side of the central ridge 211 extending from the front to the rear of the helmet 210 at the top. These ventilation holes allow airflow, along with the interior of the helmet 210, to provide a cooling effect to the user's head. See the following examples and... Figures 4A to 4EThe connection between the ventilation hole 214 and the deformable ventilation connector is described.

[0073] At the lower rear portion of the helmet 210 is an adjustable neck brace 270. The adjustable neck brace 270 is configured to rest against the back of the user's neck (e.g., at the base of the head above the neck) to help distribute the weight of the helmet 210, which has an integrated electronic subsystem 260. The adjustable neck brace 270 may include an inner neck pillow 271 and an outer rear brace portion 272. The outer rear brace portion 272 is connected to the interior of the helmet 210 via a tension member 274 and a rear connector 275. The tension member 274 may include a polymer or fabric member arranged to absorb tension (e.g., when the adjustable neck brace 270 remains stationary and the helmet 210 moves forward over the head). The rear connector 275 may include a polymer member that centrally aligns the outer rear brace portion 272 against the user's neck.

[0074] Figure 2D A rear view 203 of the helmet 210 is shown. This view shows more of the rear housing 262 and two ventilation grilles 264-A, 264-b spaced laterally at the rear. At the center of the rear housing 262 is a fan mounting 266, within which a fan drives airflow over the cooling plate of the integrated electronic subsystem 260. In use, the fan can be used to create an air pressure differential, causing air to enter through the fan mounting 266 and flow over the cooling plate, then exit via the ventilation grilles 264-A, 264-B. In an alternative example, the airflow direction can be reversed to provide a similar cooling effect. The fan arrangement keeps the integrated electronic subsystem 260 properly cooled and prevents the electronics from overheating and causing discomfort to the user. This is particularly important because the generation of augmented reality views is a resource-intensive computation that can cause high temperatures in an uncooled system.

[0075] Figure 2DTwo panels 222 of the augmented reality display are also shown: a left panel 222-B for the left eye and a right panel 222-A for the right eye. The bottom of the left wing portion 240-B and the right wing portion 240-A are also visible. Panels 222 can be electrically connected to a drive circuitry (shown in later figures) mounted in the front of the helmet 210. Each display panel 222-A, 222-B may include a waveguide for displaying an image or video frame projected by a connected microprojector. The waveguide may include a double-plate waveguide with a 40° diagonal top injection. This set of two waveguides and the corresponding projector may be referred to as an optical module or optical engine. The optical module may be driven in a manner similar to a conventional display (e.g., via a known or custom display connector and driven by a graphics unit for a computing module). A portion of the driving electronics (e.g., forming part of an embedded graphics processing unit) may be decoupled from the integrated electronic control subsystem 260 (e.g., it may be provided as part of the front circuitry 372, such as...). Figure 3C (As shown). Figure 2E This is a perspective view 204 of the front of the helmet as seen from above. The central ridge 211 and two sets of three ventilation holes 214 are visible. Figure 2F It is another perspective view 205, but this time it is from the rear of the helmet 210. Figure 2G This is view 206 of the top of the safety helmet 210. View 206 indicates the two cross sections A-A' and C-C' shown in the following figures.

[0076] Figure 2HThis is a view 207 of the underside or bottom of the helmet 210. Certain features inside the helmet 210 are visible in this view. The safety goggles 220 are visible below the brim 218 of the helmet 210. Portions of the right panel 222-A and left panel 222-b, used for the augmented reality display, are also visible. The safety goggles 220 include a nose bridge 224, which helps to support the goggles on the user's nose, thereby correctly positioning the panels relative to the user's eyes. In an alternative example, the nose bridge 224 may be positioned between the panels themselves. The underside of the rear housing 262 (e.g., the inner edge of the housing) has multiple (six in the figure) photoelectric sensors 212, and the undersides of the right wing portion 240a and the left wing portion 240b are also visible. To complement the adjustable neck brace 270, a bracket 241 is provided for positioning the helmet on the user's head. The helmet 241 rests on the front and sides of the user's head and includes a front support portion 242 and a side support portion 244. The front support portion 242 and the side support portion 244 may include pads connected to the polymer helmet frame via hook-and-loop fasteners. The helmet frame may be adjustable. For example, the helmet frame may allow at least 10 mm of vertical height adjustment, and thus allow different positioning of the helmet 210 on the user's head. In general, the user's head can be supported by the front support portion 242, the side support portion 244, and the rear support portion 272, wherein the rear support portion 272 can be adjusted independently of the front support portion 242. (Reference) Figures 7A to 7C More details about the bracket and its adjustment are described below.

[0077] Figure 2H Two electrical ports 268-A and 268-b at the rear of the safety helmet 210 are also shown. In this example, these electrical ports include a USB-C port, but can be any known electrical port. The two electrical ports 268 can be used to power the integrated electronics subsystem 260 and to transfer data to and / or from the integrated electronics subsystem 260 (e.g., for firmware or other operating software updates, for uploading BIM data, for configuration and / or for downloading mapping data). Any one or both of the electrical ports 268 may also include display ports, such as a USB-C DisplayPort (DP) or a port for DP AltMode. Therefore, ports 268 can allow virtual overlays in images or videos captured during site inspections to be viewed externally (e.g., at an on-site or off-site location with an external display such as a television screen or monitor). The display output provided by one or more of the ports 268 can utilize graphics acceleration and / or augmented reality capabilities for displaying virtual images on the built-in display panel 222, i.e., the functionality of the integrated electronics subsystem 260. In one scenario, electrical port 268 can also charge a removable rechargeable battery currently mounted within wing portion 240 (but this may not be provided in all implementations).

[0078] When wearing the helmet 210, the user's head can be supported by the internal woven mesh 248 (in Figure 2H (See in the figure below and also shown in the accompanying drawings). The woven mesh 248 may include a fabric mesh that snaps into the side of the helmet's interior. The woven mesh 248 may be fire-resistant and / or impact-resistant. In a preferred embodiment, the woven mesh 248 includes an impact-resistant digitally knitted comfort headband that is secured to the interior of the helmet 210 with a pointed end.

[0079] Example of a double-layered safety helmet

[0080] Figures 3A to 3D The construction of safety helmets (such as) is shown Figures 2A to 2H An example of a safety helmet (210). In this example, the safety helmet with an integrated electronic subsystem includes an outer part and an inner part. Figure 3A This is a perspective side view 300 of an example outer part 310 of a safety helmet. Then, Figure 3B This is a perspective side view 302 of an example internal portion 332 of a safety helmet. The outer portion 310 and the inner portion 332 are spaced apart within the safety helmet; for example, the safety helmet is configured such that a gap exists between the outer portion 310 and the inner portion 332. This gap can be approximately 20 mm. In this example, an integrated electronic subsystem is mounted between the outer and inner portions. For example, the integrated electronic subsystem could be mounted... Figure 3B Position 360. In some cases, impact foam 380 may be included to increase protection; however, in other cases, impact foam 380 may be omitted. The integrated electronic subsystem may be mounted on one or more of the external and internal portions (e.g., screw bosses may be provided on one or more of said portions to allow screw attachment of one or more printed circuit boards and / or computational packages).

[0081] In a preferred example, both the outer and inner portions are rigid, including, for example, a "shell" protecting the user's head. This dual-layer system provides secure protection for the user's head while reducing weight on the head and providing a structure that can safely accommodate the integrated electronic subsystems. Maintaining low weight is particularly important because the integrated electronic subsystems and other electronic components increase the weight of conventional protective helmets.

[0082] Go to Figure 3A The external portion 310 shown here can provide Figures 2A to 2H The outer portion of the safety helmet 210 shown. As described with reference to those figures, the outer portion 310 has holes 312 for mounting photoelectric sensors (e.g., 212) and ventilation holes 314 (e.g., as shown in the figures). Figure 2E(As shown in 214). The outer portion 310 may include a polymer shell. For example, the outer portion 310 may include a 1.5 mm thin protective polymer shell, similar to the polymer shell of a helmet without integrated electronic subsystems. In some cases, the outer portion 310 may be thinner than the contrasting helmet because impact protection is distributed across both shells. The polymer shell may be molded with holes 312 and 314 and a visor 318. Figure 3A The rear portion 362 shown may be provided as a separate part of the polymer molded housing (e.g., as explained above with reference to rear housing 262) or as an integrated part. In some cases, the main outer portion 310 and the rear portion 362 are molded separately and then joined together during manufacturing.

[0083] Go to Figure 3B In this example, the inner portion 332 includes a rigid inner section that provides protection for the user's head. During construction, an integrated electronic subsystem is mounted at position 360 at the rear of the helmet, and then the inner portion 332 and the outer portion 310 are engaged and fastened together for use. In the example shown, the outer portion 310 includes screw bosses that align with complementary screw holes (e.g., through holes) on the inner portion, such that the inner portion 332 is screwed onto the outer portion 310 to engage them together. Figure 3B A set of mounting screw bosses for the front cover portion (i.e., including the safety goggle 220) is also shown. In this example, the two portions can be separated for repair and maintenance, such as to access the integrated electronic subsystem. In some cases, a seal can be formed between the outer and inner portions (e.g., via silicone sealant and / or rubber O-rings or other sealing elements) to prevent water from entering the gap between the portions. Although screws are shown here, other fastening and / or connection methods may be used in other examples.

[0084] In a preferred embodiment, the inner portion 332 comprises a carbon fiber inner shell. Typically, the outer portion may comprise a polymer outer shell having a first thickness, and the inner portion may comprise a carbon fiber inner shell having a second thickness. Preferably, the second thickness is less than the first thickness. The first thickness may be 1.5 mm, and the second thickness may be 0.8 mm. The inner portion can be formed by hot-pressing carbon fiber sheets. The carbon fiber inner shell helps reduce the weight of the helmet while maintaining impact protection. For example, the carbon fiber inner shell can provide most of the impact and penetration protection, while the outer polymer shell can serve as a decorative outer shell and deflect the initial portion of the impact energy (e.g., provide the remaining impact absorption).

[0085] like Figure 3BAs shown, a set of ventilation holes 334 may be provided in the internal portion 332. These internal ventilation holes 334 are aligned with the external ventilation holes 314. These may be directly connected (e.g., simply by aligning the holes on the two portions), connected via a rubber seal, or preferably via a deformable ventilation coupling, as shown in the reference. Figures 4A to 4E A more detailed illustration and description follows. Holes for ventilation and / or fastening screws can be cut within the internal portion 332 using a computer numerical control (CNC) laser cutting machine. A CNC cutting machine can also be used to finish features such as the inner cornice 338. Figure 3B The image shows a left battery mounting unit 340-B, which receives a removable battery encapsulated within a removable battery housing (e.g., corresponding to...). Figure 2C (Wing 240-B in the middle).

[0086] Figure 3C It shows the following: Figure 2G A first cross-sectional view 304 along section line A-A' is shown. Parts of the safety glasses (e.g., 220), augmented reality display panel (e.g., 222), and camera assembly (e.g., 230) are visible at the front 320 of the helmet. A removable battery can be considered as being mounted using wings 340-B and a bracket 355 for adjusting the helmet's vertical height. An adjustable neck brace 370 (e.g., corresponding to 270 above) is also shown.

[0087] exist Figure 3C In this view, the outer portion 310 and the inner portion 332 are visible. Furthermore, the connection between the outer vent 314 and the inner vent 334 is shown. In this view, the integrated electronic subsystem 360 is visible at the rear of the helmet, as if mounted in the gap 336 between the outer portion 310 and the inner portion 332. Figure 3C An integrated electronic subsystem 360 is shown, comprising at least one circuit board 367 with mounted processing electronic devices and a heat sink 368 with cooling fins. In use, a fan 366 drives air above the heat sink 368 to cool the electronic devices (e.g., as shown in the reference). Figure 2D (As described in 264 and 266). The integrated electronic subsystem 360 can be isolated from the internal spacer 336, for example, it can be mounted to the rear of the helmet as part of a hermetically sealed molded package. Figure 3C Also shown is a woven support mesh 348 (e.g., equivalent to...). Figure 2HThe 248 in the middle can be fastened to multiple sets of tips 349. At least four sets of tips 349 may be present, each set including multiple (e.g., 3) individual tips for holding corresponding loops and / or gaps in the fabric of the woven support mesh 348. Each set of tips can be fastened to the inner portion 332 via holes cut or drilled in the inner portion. Therefore, the woven support mesh 348 can be removed for regular washing and cleaning (e.g., in a washing machine or in water containing detergent).

[0088] Figure 3C A front circuit 372, also mounted between the external portion 310 and the internal portion 332, is also shown. The front circuit 372 may include electronics for driving an augmented reality panel (such as 222) and / or for receiving data from a front camera assembly (such as 230). In one case, the front circuit 372 may include a driver board for a portion of the optical assembly used to render an augmented reality image (e.g., on the display panel 222). In some cases, for example, the front camera assembly (such as 230) may be communicatively coupled (e.g., via a flexible cable) to a printed circuit board forming part of the integrated electronic subsystem 360, rather than communicatively coupled to the front circuit 372. In some cases, in addition to the integrated electronic subsystem, the front circuit 372 may include a printed circuit board (PCB) or chipset to provide specific functionality. For example, the front circuit 372 may include a wireless communication module (e.g., a 2.4 GHz or 5 GHz wireless communication module) for communicating with peripheral devices and / or other devices. The wireless communication module can be used for data communication with a handheld controller. The front circuitry 372 may include one or more application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs). In this example, the front circuitry 372 is mounted to a screw boss located in the outer portion 310. The spacing 336 between the inner and outer portions may be approximately 20 mm, but can vary on the helmet, for example, increasing from approximately 20 mm at the front to approximately 25 mm at the rear. Typically, the outer and inner portions may be spaced approximately 20 mm apart for at least half the circumference of the helmet.

[0089] Typically, an outer and inner section can provide a protective, waterproof, sealed chamber to house a complex set of electronics while maintaining the head-mounted product's low weight. The two-layer design can provide protection according to building safety standards (e.g., American standards set by ANSI (American National Standards Institute) – including Type I protection – ANSI Z89.1, British standards set by BSI – BS, European standards – EN, such as BS EN 397, and / or international standards from ISO – among others). The carbon fiber inner section provides a lightweight primary protective shell for one or more impact and penetration protection components, allowing for the mounting of electronics, and is then supplemented by an upper or outer polymer protective shell that defines the appearance and provides part of the external sealing and impact protection. The outer polymer protective shell can provide protective impact absorption.

[0090] Figure 3D It shows the following: Figure 2G The second cross-sectional view 306, shown along section line C-C', shows the outer portion 310 and the inner portion 332, which are separated by a spacer 336. In some configurations, the spacer 336 may be at least partially filled with impact foam 380. The impact foam may include closed-cell foam, expanded polystyrene (EPS), and non-Newtonian polymers (including so-called "smart" foams engineered to provide specific impact absorption properties, such as those marketed under the trade name "D3O"® by Design Blue Limited). Thus, in some configurations, the helmet is provided with usefully spaced outer and inner (rigid) portions, with impact foam arranged in the spacer between the outer and inner portions. The impact foam 380 can improve the absorption of energy from one or more vertical and lateral impacts. The impact foam 380 can be particularly beneficial for lateral impacts (i.e., from the side of the helmet or at an angle to the helmet, including from the front, rear, and lateral sides). Utilizing a double-layer design, the impact foam 380 can be easily installed between the inner portion 310 and the outer portion 332, while concealing the impact foam during use so as not to be seen and / or damaged. This maintains the helmet's compact shape. The use of the impact foam shown in the figure allows the helmet to meet the ANSI Type II standard requiring side impact absorption. Furthermore, the impact foam does not affect the comfort of the helmet wearer or compromise the helmet's structure. The impact-resistant foam sandwiched between multiple shell layers provides a certified AR-constructed helmet for side impacts (where "side impact" includes impacts from generally horizontal forces on one or more of the front, rear, and sides of the helmet, for example, opposite to impacts from above).

[0091] Example of a ventilation connector

[0092] Figures 4A to 4E Provides references to the above. Figures 3A to 3D Examples of ventilated couplings used in conjunction with the described double-shell design. The ventilated coupling includes a connection between the inner and outer portions of the helmet (e.g., as described above), where the inner portion may include a carbon fiber shell and the outer portion may include a polymer shell. In a preferred example, the ventilated coupling includes a deformable ventilated coupling for connecting the outer and inner portions, wherein the deformable ventilated coupling allows air to flow from vents in the inner portion to the outside of the outer portion (and / or vice versa). In some examples, the deformable ventilated coupling provides a measure of impact absorption and reduces energy transfer from the impact to the inner portion and then to the user's head. The deformable ventilated coupling can be provided as a replaceable part, for example, it can be replaced as part of the helmet's repair and maintenance. In most cases, the replaceable part is replaced by the manufacturer or a certified dealer (e.g., not the user, as this may be unsafe) who can ensure the safe operation of the helmet. If the helmet is impacted, it is recommended to return it and not use it anymore. In one embodiment, the deformable ventilated connector includes a first rigid frame for attachment to an internal portion of the helmet, a second rigid frame for attachment to an external portion of the helmet, and a deformable suspension system disposed between the first and second rigid frames. The deformable suspension system includes openings allowing air to flow from vents in the internal portion to the exterior of the external protective portion, the openings including waterproof seals. The deformable ventilated connector improves shock absorption while also improving cooling and user comfort.

[0093] Figure 4A An exploded view 400 shows an outer portion 410 and an inner portion 432 of a safety helmet. The outer portion 410 and the inner portion 432 may include, as referenced above... Figures 3A to 3D The external portion 310 and the internal portion 332 are described. Figure 4A The helmet may include an outer portion 410 of a polymer shell having a series of external ventilation holes 414 for allowing air to flow from the interior of the outer portion to the exterior. In this example, the holes are laterally arranged on the central ridge of the helmet, with a series of three elongated holes extending from front to back along each side of the central ridge. The inner portion 432 also includes a corresponding set of internal ventilation holes 434, which allow air to flow from the lower side of the inner portion 432 to the upper side of the inner portion 432. In this example, the internal ventilation holes 434 comprise two rows of three elongated holes parallel to each other, aligned with the external ventilation holes 414. In one case, the internal ventilation holes 434 may be CNC machined or drilled into the carbon fiber shell.

[0094] Figure 4AA deformable ventilation coupling 450 is also shown, installed between the inner portion 432 and the outer portion 410. The deformable ventilation coupling 450 forms a seal between the inner ventilation opening 434 and the outer ventilation opening 414, thereby allowing air to flow from the interior of the inner portion 432 to the exterior of the outer portion 410 (and vice versa). Figure 4A In this configuration, the internal ventilation opening 434 includes a rubber bushing to facilitate sealing and engagement with the deformable ventilation coupling 450. Figure 4A In one example, the deformable ventilated coupling 450 includes a first rigid frame 452 for coupling to an inner portion 432, a second rigid frame 454 for coupling to an outer portion 410, and a deformable suspension system 456 disposed between the first and second rigid frames 452 and 454. The first rigid frame 452 may include a molded polymer mount having holes or bosses to allow fastening to the inner portion 432, for example, via screw holes 435 protruding from the surface of the inner portion, thereby allowing the first rigid frame 452 to be screwed onto the inner portion 432. In other cases, a boss or tip may be provided including an insert with small molded (or pre-drilled) holes in the inner portion 432. Figure 4A In the first rigid frame 450, there are four screw holes 435 at the four corners of the area surrounding the internal vent 434. Similarly, the second rigid frame 454 may also include a molded polymer frame with holes to allow fastening to the outer portion 410. The outer portion 410 may include bosses that allow the second rigid frame 454 to be screwed onto the outer portion 410. For example, screws can fasten protruding holes 462 to the inside of the outer portion 410 (i.e., holes 462 protruding from the second rigid frame 454 in this example). Although removable mounting is preferred for repair and replacement (e.g., under manufacturer's factory conditions), in alternative examples, the deformable ventilator 450 may be permanently fastened via welding, polymer overmolding, or adhesive. During manufacturing or repair, the deformable ventilator 450 is attached to the inside of the outer portion 410 (as indicated by the arrow above), and then the inner portion 432 is aligned such that the screw holes 435 mate with the underside of the first rigid frame 452 before the inner portion 432 is fastened to the outer portion 410 (e.g., as shown in the reference). Figure 3B As described, via a screw, through a hole in the inner portion and a screw boss in the outer portion.

[0095] Figure 4B A top view 402 of the safety helmet is shown, indicating section line A-A' and external ventilation holes 414 on the outer portion 410. Figure 4C A cross-sectional view 404 along the cross-sectional line A-A' is shown. Figure 4C The deformable ventilation coupling 450 is shown installed and can be considered as being related to the above. Figure 3CComplementary. In Figure 4C In the diagram, we can see how the deformable suspension system 456 forms a series of channels 458 connecting the internal vent 434 and the external vent 414. A series of channels 458 are formed within a gap 436 between the external portion 410 and the internal portion 432, i.e., the same gap that allows for the installation of the integrated electronic subsystem 460. The channels 458 may be provided with waterproof seals (e.g., at the bottom of the channel near the internal vent 434 or at the opening of the external vent 414) that prevent water ingress but allow airflow (e.g., in the form of a breathable but waterproof membrane). The channels may widen toward the external vent 414 such that air from the sides of the impact center ridge is directed into the helmet to cool and ventilate the user's head.

[0096] Figure 4D A deformable ventilated connector 450 is shown as a separate component of the helmet. For example, the deformable ventilated connector 450 may be supplied as a spare part. Figure 4D A first rigid frame 452 and a second rigid frame 454 are shown connected by a deformable suspension system 456. The deformable suspension system 456 may include a rubber member that absorbs energy from an impact and reduces the amount of energy transmitted to the inner portion 432. For example, upon impact, the deformable suspension system 456 can be compressed (i.e., deformed) in a substantially vertical direction between the inner portion 432 and the outer portion 410. The deformable suspension system 456 forms a series of channels 458 to provide airflow between external vents 414 in the outer portion 410 and internal vents 434 in the inner portion 432. This... Figure 4E It is shown in more detail in the sectional view.

[0097] therefore, Figures 4A to 4EA safety helmet is shown, comprising an outer section, an inner section including vents, and a deformable ventilated connector for connecting the outer and inner sections. The deformable ventilated connector allows air to flow from the vents to the outside of the outer section. The deformable ventilated connector can be used with any multi-layered safety helmet, but is particularly useful for helmets with integrated electronic subsystems, where the integrated electronic subsystems are installed between the inner and outer sections, and the inner section forms a base worn by the user, for example via a padded support and an internal woven mesh. A rubber suspension system in a construction safety helmet can provide utility as part of a ventilation, waterproofing, and shock absorption system in a double-shell design. The double shell achieves lightweight and thin material walls and waterproofing of the internal spacer, while the deformable ventilated connector provides energy absorption and waterproof airflow. The flexible, deformable suspension system ensures absorption of any impact, unlike a rigid, sealed ventilation system where impact energy is directly transferred to the wearer's head. This system creates a series of sealed channels that allow airflow (i.e., the entire helmet can breathe) without wetting the user or allowing dust to enter. Deformable ventilated connectors provide a seal vertically (e.g., via breathable and waterproof components) and horizontally relative to the spacing between layers inside the helmet.

[0098] Example battery connection interface and removable housing portion

[0099] Figures 5A to 5G An example component for a battery connection interface of a safety helmet with an integrated electronic subsystem is shown. The safety helmet includes multiple battery connection interfaces for connecting a plurality of corresponding removable batteries. This then enables a "hot-swappable" function, wherein the power subsystem of the integrated electronic subsystem is configured to draw power from one of the multiple removable batteries connected, allowing the replacement of another of the multiple removable batteries without power loss to the integrated electronic subsystem. This then allows the safety helmet to be used for extended periods in the field (e.g., on a construction site) without downtime for battery replacement. This is particularly beneficial for augmented reality devices, as downtime may require recalibrating the system to view augmented reality images, and the power loss from battery replacement results in the erasure of volatile memory.

[0100] Figures 5A to 5G An example battery connection interface is further illustrated, providing a beneficial release and battery housing system. By using a removable battery housing including a securing mechanism for holding the removable battery within the housing, the battery can be easily removed and replaced without loss or drop. This again further promotes the use of safety helmets in the "field" of construction sites.

[0101] Figure 5AA side perspective view 500 of an exemplary safety helmet 510 is shown. The safety helmet 510 may include, as shown in any previous drawing (e.g., Figures 2A to 2E , Figures 3A to 3D and / or Figures 4A to 4E The safety helmet 510 is shown and described above. The safety helmet 510 includes an integrated electronic subsystem 560, which, as previously described, may include at least one processor and memory, such as a computing module. The computing module may form part of an augmented reality system for displaying augmented reality images on a display panel 522 mounted behind a set of safety goggles 520. The integrated electronic subsystem 560 may have features similar to previous examples (e.g., 260, 360, 460), and similarly, the AR display panel 522 and the safety goggles 520 may be as described in reference... Figures 2A to 2H (For example, configurations as described in 222 and 220)

[0102] Figure 5A An example with two battery connection interfaces 505 is shown. The battery connection interfaces 505 are arranged together with the helmet 510 and allow a removable battery 550 to be received to power the integrated electronic subsystem 560. The battery connection interfaces 505 are mounted laterally within the helmet 510, for example, on either side of the helmet near the wearer's ear. The removable battery 550 may include a rechargeable battery, such as a lithium-ion rechargeable battery. The battery connection interfaces 505 include a combination of mechanical and electrical interfaces to receive the removable battery 550 carried by one of a set of wings 540. Figure 5A The diagram shows the right wing portion 540-A removed from the helmet 510 and the left wing portion 540-B retained within the helmet 510. Each wing 540 forms the lower portion 534 of a removable battery housing, which, in use, forms part of the side of the helmet, extending the edge 524 of the visor 520. Figure 2A , Figure 2C , Figure 2E and Figure 2F The image shows an example of the lower portion 534 of a removable battery housing, visible as wings 240-A and 240-B. Specifically, Figure 2C This illustrates how the upper portion 536 is not visible when used with a battery mounted in the helmet. The upper portion 536 is housed within a retaining housing 544. The retaining housing 544 may include a molded polymer housing (e.g., as shown in the image) attached to the interior of a multi-layered helmet. Figure 3B and Figure 4A(As shown). In other examples, the fixed housing 544 may form a molded portion of a single-layer polymer shell or an insert of a single-layer shell. In the examples herein, the wings shown as 240-A and 240-B can be considered to form flanks of a viewing component for an augmented reality system. The viewing component for an augmented reality system may include one or more components that enable a user to view augmented reality images, including so-called mixed reality and virtual reality as previously described. The viewing component may include one or more of a safety goggle 520 and an internal display panel 522.

[0103] Figure 5A A safety helmet 510 is shown during a battery replacement operation. In this operation, the user replaces the right battery 550-A with a new or newly charged battery. In this arrangement, the integrated electronic subsystem 560 is powered by a battery (not shown) contained within a removable battery housing that forms the mounting of the left wing portion 540-B. The user can begin with both batteries installed and both wing portions in place (e.g., as shown). Figure 2A , Figure 2C , Figure 2E and Figure 2F(As shown). In this state, the user can be notified that one of the batteries is low (here, the right battery 550-A). This can be done, for example, via a user interface shown on the augmented reality display panel 522 (e.g., as part of a head-up display HUD) and / or via indicators (such as LEDs) on the helmet and / or handheld controller. The charging or power supply status or level can be indicated via the aforementioned method. While the user is still using the augmented reality display panel 522 to view augmented reality images, the user can initiate the battery replacement operation by pressing the release button 548-A at the rear of the wing portion 540-A. The release button 548-A mechanically releases the removable battery housing from a battery housing interface that forms part of the helmet (e.g., from interface 505-A, which is partially formed by the retaining housing 544). The user can also use the front grip portion 542 to pull the removable battery housing to assist in its removal. With the design of the helmet 510, once the release button 548-A is pressed, the weight of the battery 550 in the removable battery housing also provides a force under gravity, causing the removable battery housing to move downwards naturally. In the current configuration, the release button 548-A forms the lower portion of a spring-loaded pivot latch. When installed, the upper portion 549-A of the pivot latch rests on a tab within the inner portion (i.e., a tab forming part of the mechanical interface on the helmet). When the release button 548-A is pressed, the upper portion 549-A of the pivot latch is released, thereby releasing the latch from the tab; for example, the upper portion 549-A moves to be flush with the rear surface of the wing 540-A, allowing the wing 540-A to move downwards out of the helmet. Once the removable battery housing is released, the battery can be removed from the housing and replaced with a charged battery (e.g., as described below). Figure 5B(As described above). The removable battery housing can then be returned to engage with the corresponding battery connection interface 505-A on the helmet. For example, if battery 550-A represents a new, fully charged battery, an upward force can be applied to the wing 540-A to "lock" the upper portion 536 of the removable battery housing into the corresponding interface on the helmet. For example, when the release button 548-A is not pressed, the base of the upper portion 549-A is pushed outward, for example by spring loading. As the removable battery housing is pushed upward into the corresponding interface, the base of the upper portion 549-A pivots inward against the thrust as the upper portion 549-A contacts the corresponding interface on the helmet, and then, when mounted in the resting position, pivots outward under the thrust to engage with the recess in the battery connection interface 505-A of the helmet (e.g., the lower edge of the base of the upper portion 549-A protrudes from the removable battery housing and rests on the upper edge of the recess forming part of the tongue). By pushing the wing 540-A back into the interface 505-A to "lock" it in place, the battery's electrical terminal 552-A mates with the corresponding electrical terminal in the helmet's battery connection interface 505-A, so that the battery 550-A can then be used to provide power to the integrated electronic subsystem 560.

[0104] Although the battery replacement procedure with a helmet worn during replacement has been described above, the battery can also be replaced with the helmet removed, for example, by inverting the helmet onto a surface and releasing the removable battery housing. Both batteries can also be replaced during a power outage. In another configuration, the batteries can be replaced via one or more of electrical ports 568-A or 568-B (e.g., according to...). Figure 2H The USB-C connector (268 in the middle) electrically connects the integrated electronic subsystem 560 to an external battery pack or power source. Power can be supplied via one or more rear electrical ports, allowing the removal of both removable battery housings while maintaining power supply to the integrated electronic subsystem 560.

[0105] Figure 5B and Figure 5C Views 502 and 504 provide views of the right-side removable battery housing 546-A removed from the helmet 510. Views 502 and 504 show how to insert and remove the battery 550-A into the removable battery housing 546-A. A similar structure and function apply to the left-side removable battery housing (but mapped symmetrically).

[0106] exist Figure 5B In the middle, battery 550-A is removed from the removable battery housing 546-A on the right side. As previously mentioned, battery 550-A can be a rechargeable battery. In this case, battery 550-A can be removed for use. Figure 10A and Figure 10BThe battery 550-A is a rechargeable battery that can be used by the component kit that includes and accompanies the helmet 510. For example, the battery 550-A can also be used to charge the helmet. Figures 8A to 8D Handheld controller and Figures 9A to 9C The tracking beacon is powered. The use of a common rechargeable battery design for multiple components facilitates use on construction sites, where batteries may need to be quickly replaced on-site and power outlets may be limited. Figure 5B In the image, battery terminal 552-A is visible at the upper end of battery 550-A. Battery terminal 552-A includes an electrical interface configured to form an electrical connection with a corresponding electrical interface (i.e., battery socket) within battery connection interface 505. Figures 8A to 8D Handheld controller and Figures 9A to 9C The tracking beacon provides a similar corresponding electrical interface and battery connection interface.

[0107] In use, battery 550-A is inserted into the interior 554-A of the right-side removable battery housing 546-A. Figure 5C The battery 550-A is shown in its proper position within the removable battery housing 546-A on the right side. See below for reference. Figures 5D to 5G In a more detailed description, the removable battery housing 546 may include a securing mechanism to hold the removable battery within the removable battery housing when it is not attached to the helmet. Figure 5C In this configuration, the fixing mechanism holds or clamps the battery 550-A within the removable battery housing 546-A on the right side. Therefore, as... Figure 5C The right-side removable battery housing 546-A, which includes battery 550-A, can be inverted without the battery 550-A falling out (e.g., sliding out of the interior 554-A). In addition to a release button 548-A that allows the right-side removable battery housing 546-A to mechanically separate from the helmet, the right-side removable battery housing 546-A also includes a mechanical latch 549-A. Pressing the battery release switch 548-A releases the mechanical latch 549-A, allowing it to pivot inward and the right-side removable battery housing 546-A to be removed from the helmet.

[0108] Figure 5D A rear view 506 of the right-side removable battery housing 546-A is shown. This shows the helmet release button 548-A and the mechanical latch 549-A, as well as the protrusion 588-A of the pivot member 581-A. Figure 5D The section line E-E' is also shown. Figure 5E The cross-sectional view 508 shows the cross-section along line E-E'. Figure 5EThe interior of the right-side removable battery housing 546-A is shown. Similarly, a similar design applies to the left-side removable battery housing, allowing for symmetry. It should be noted that in this example, each removable battery housing 546 is configured to accommodate a common (i.e., identical) battery design. Therefore, minor differences in the interface design may exist to receive the same battery on both the left and right sides (e.g., the terminal connectors rotate or are configured to receive rotating batteries).

[0109] Figure 5E The battery 550-A is shown installed together with the removable battery housing 546-A on the right. Figure 5E A securing mechanism 580 is also shown that holds the removable battery within the removable battery housing when it is not attached to the helmet. While the securing mechanism 580 could be used solely to secure each battery within its corresponding battery housing, the securing mechanism 580 in this example has the additional feature of being released when the removable battery housing is installed within its corresponding battery connection interface. About Figure 5G This is illustrated in more detail. Thus, in this example, the securing mechanism 580 is configured to secure the removable battery within the removable battery housing when the removable battery housing is not attached to the helmet, and is arranged to release the removable battery when the removable battery housing is attached to the helmet via a mechanical interface forming part of the battery connection interface 505.

[0110] Go to Figure 5E The fixing mechanism 580 shown in this example includes a clamping mechanism for applying frictional force to a battery 550-A inside a removable battery housing. This clamping mechanism includes a pivoting member 581-A having a central pivot 584-A. A protrusion 588-A forms the upper end of the pivoting member 581-A, which protrudes from the removable battery housing. The lower end of the pivoting member 581-A includes a force-applying member 586-A, which is pushed toward the battery 550-A by a pushing member. In this example, the pushing member includes a coil spring within the removable battery housing, but alternatively includes a leaf spring or a small electromechanical device. When the removable battery housing is not installed inside the helmet 510, the pushing member biases the force-applying member 586-A toward the battery 550-A. Figure 5EIn this example, the force-applying member 586-A also includes a friction pad 587-A, which, when installed within the removable battery housing, contacts the exterior of the battery 550-A, increasing friction between the battery 550-A and the force-applying member 586-A. In this example, the combination of natural surface friction experienced by the battery 550-A within the interior 554-A of the removable battery housing and the additional frictional force applied by the force-applying member 586-A is greater than the downward force due to gravity caused by the weight of the battery 550-A. Therefore, when the removable battery housing is inverted with the battery inside, the battery will not fall out of the removable battery housing. To release the battery 550-A, the user can simply pull out the battery and overcome the frictional force. In some embodiments, the user may alternatively press the protrusion 588-A to pivot the force-applying member 586-A away from the battery. This disrupts the contact of the friction pad 587-A and allows the user to pull up the battery 550-A or shake it out when it is upside down or otherwise tilted downwards. In a preferred example, the friction pad 587-A comprises a thermoplastic polyurethane-TP-grip, which provides good durability and abrasion resistance.

[0111] Figure 5E The removable battery housing shown also includes a battery biasing member 590-A located at the bottom of the interior 554-A of the housing (within the wing 540-A). The battery biasing member 590-A applies a force to the base of the removable battery 550-A. When the battery 550-A is within the removable battery housing and not installed within the helmet 510, the force applied by the force-applying member 586-A is greater than the force applied by the battery biasing member 590-A, and therefore the battery is not pushed upwards within the removable battery housing. However, as Figure 5F and Figure 5G As shown, when the removable battery housing is attached to the helmet 510, the force-applying member 586-A is disabled, thereby allowing the battery biasing member 590-A to apply an upward force to the removable battery 550-A to facilitate electrical connection within the battery connection interface 505-A, that is, to form an electrical connection between the removable battery 505-A and the integrated electronic subsystem 560 of the helmet 510.

[0112] Figure 5F and Figure 5G Views 512, 514, and 592 provide further details showing the battery connection interface 505 within the helmet 510. In these examples, the helmet 510 includes, for example, reference... Figures 3A to 3D The described double-layered, double-shell safety helmet; however, similar structures and functions can also be used for single-layered safety helmets. Figure 5F Rear view 512 and side view 514 of the internal portion 532 are shown. The internal portion 532 may include a carbon fiber shell. The carbon fiber shell may have a thickness of approximately 0.8 mm, as shown in the reference. Figures 3A to 3D As described above. In views 512 and 514, the internal portion 532 is shown without the integrated electronic subsystem 560. In side view 514, the mounting location of the integrated electronic subsystem 560 is indicated by arrow 562. View 512 shows the cross-sectional line C-C', and view 514 shows the cross-section C-C' passing through the battery connection interface. Figure 5G The text shows area 592 in more detail.

[0113] Figure 5G The region 592 is shown as a cross-section passing through the battery connection interface 505-A. Figure 5G A battery 550-A is shown mounted within a right-side removable battery housing 546-A, and the right-side removable battery housing 546-A is coupled to a battery connection interface 505-A. A retaining housing 544-A forms part of a mechanical interface that holds the right-side removable battery housing 546-A within a safety helmet 510. When the right-side removable battery housing 546-A is inserted, a portion 545-A of the retaining housing 544-A applies a force to a protrusion 588-A of a pivoting member 581-A, causing the lower end of the pivoting member 581-A to pivot away from the surface of the mounted battery 550-A. This allows the battery biasing member 590-A to push the base of the battery 550-A upward into the battery connection interface 505-A. This then ensures a good electrical connection between the battery terminal 552-A and the corresponding electrical interface 594 of the battery connection interface 505-A. This is advantageous on construction sites, where dust, impacts, and movement can easily damage electrical connections. The electrical interface 594 may include a printed circuit board (PCB) connector. Therefore, the combination of the securing mechanism 580 and the battery biasing member 590 prevents the battery from being lost or damaged during replacement and also ensures a secure connection after installation into the safety helmet 510. This is particularly important for augmented reality systems where power loss can interrupt the augmented reality view and necessitates timely recalibration and initialization.

[0114] therefore, Figures 5A to 5GAn example of an internal hot-swappable battery for a safety helmet with an integrated electronic subsystem is shown, wherein the subsystem may include a computing module for an augmented reality system. The integration of multiple battery connection interfaces and multiple corresponding removable batteries within the safety helmet means that an augmented reality view can be provided during battery replacement, enabling continuous 24-hour use of the augmented reality device. While described with reference to a construction environment, it should be noted that in other examples, the hot-swappable battery arrangement can be used in augmented reality devices used outside the construction site; for example, the method described herein can be applied to augmented reality head-mounted devices that are not safety helmets. However, there are specific advantages designed for the limitations and challenges encountered on construction sites.

[0115] The described example of an internal hot-swappable battery in a helmet with an integrated electronic subsystem uses a hot-swappable battery housed in a snap-on, removable shell wing structure, which results in integration into the helmet (and more generally, into augmented reality devices). The battery can be removed with one hand when the head-mounted device is worn, or easily when the unit is not worn.

[0116] The described example provides advantages over contrasting examples with fixed internal batteries or where the battery needs to be carried separately by the user (e.g., in a backpack or on a belt clip). In the latter case, the integrated, removable battery housing frees up space on the user's body, which is important for busy and often confined spaces on construction sites. It is also safer because there are fewer wires and cables snagging on objects. It also overcomes the problem of battery capacity loss or degradation because the battery is easy to replace. This is a collaborative feature when multiple components in an augmented reality kit share the same removable, replaceable, rechargeable battery, as a set of batteries can be charged for a day on site and set up for a constant charging rotation (e.g., in an office on site, etc.). If a battery exhibits poor performance or a problem, it can be easily removed from the cycle and replaced with a new battery without concern for compatibility or type.

[0117] The described example also provides advantages over contrasting examples that require the electronic or augmented reality device to be plugged in and / or taken offline for charging. This can be time-consuming and prevents users from using the device while it is charging. However, this example allows for uninterrupted power throughout the day—multiple hot-swappable batteries allow users to replace the batteries on the device without interrupting power.

[0118] The exemplary securing mechanism described herein also prevents the user from accidentally dropping or mishandling the battery during battery replacement operations, a risk that can occur on busy and restricted construction sites. Therefore, this example provides a battery holding system that retains the battery within the removable wing structure when it is removed from a helmet or other augmented reality device. This provides the ability to keep the battery within the removable wing housing when the housing is removed from the respective device (such as a helmet) at various angles (including during use or when the device is inverted on a table or in hand).

[0119] In a preferred example, a spring-loaded force-applying member or "clamp" provides frictional retention on one side of the battery when the removable battery housing is removed from the device. When the removable battery housing is inserted into the device, the clamp is released, allowing a positive electrical connection between the battery and the corresponding circuit connector. This ensures a reliable electrical connection.

[0120] In the described example, the battery can be removed from the augmented reality helmet on the user's head, or placed upside down on a surface at any angle. A securing mechanism prevents the battery from falling out of the casing when inverted. This capability is provided without compromising the robust positive connection between the battery and the helmet when the battery is stored inside.

[0121] Figure 6 This is a side view 600 of the internal portion 632 of an exemplary safety helmet, illustrating how the location of a set of battery connection interfaces also allows for control of the helmet's center of gravity. Therefore, the center of gravity can be positioned in a manner conducive to user comfort (e.g., regarding the weight of the helmet), so that the helmet is optimally supported by the user's head. This design thus prevents negative strain from being applied to sensitive areas such as the neck. Although a two-layer, double-shell design is described with reference to this... Figure 6 This is an example, but it should be noted that the location of the described battery connection interface can also be applied to other designs, such as single-layer helmets.

[0122] Figure 6 An interior portion 632 is shown, comprising a safety goggle 620, a camera assembly 630, wings 640, and an adjustable neck brace 670. Therefore, this example may include something similar to at least [reference needed]. Figures 2A to 2H and Figures 3A to 3D The characteristics of those features described. Figure 6 Also shown is a deformable ventilation coupling 650, which can be configured according to a reference. Figures 4A to 4E The ventilation connector 450 is described. Figure 6 It is also shown that they are in the inner part 632 or the outer part (such as...) Figure 3AMultiple electronic components are mounted on the device at positions corresponding to the mounting positions on the device (310). These include a photoelectric sensor 612 mounted on an external part (e.g., in...). Figures 2A to 2H The external components shown are a photoelectric sensor 212, a rear circuit board 668 for integrating an electronic subsystem 660 (which may be configured according to subsystems 260 to 560), a front circuit board 672, and a heat sink 674. Figure 6 Mounting plate 613 is also shown. Mounting plate 613 includes a screw boss that allows for the installation of mesh fastening tips for weaving inner meshes (e.g., impact fabric meshes). Figure 3C The value shown is 349 and in Figure 7A (Shown as 762). Mounting plate 613 can be fastened into drilled or cut holes in internal portion 632 and / or attached using adhesive. Heat sink 674 includes cooling fins that allow components of rear circuit board 668 (e.g., at least one processor, memory, and other chips and / or electronic components) to be cooled, for example via, as referenced at least Figure 2D The fan operation is described in features 264 and 266. The front circuit board 672 may include components for an AR display panel (e.g., Figure 2D The electronic components include 222), camera assembly 630, and driving circuits for one or more of the photoelectric sensors 612. Different configurations and connections may exist between the electronic components.

[0123] exist Figure 6 In this configuration, the battery connection interface is housed within a fixed housing 654, which is secured to the internal portion 632. The wing 640 can be inserted into the fixed housing 654, at least as shown below. Figure 5A and Figure 5GAs shown. In this example, the battery connection interface is located laterally on the helmet, such that the helmet's center of gravity 642 is aligned with the head and neck of the user wearing the helmet. For example, the main weight of the helmet includes, for example, the integrated electronics subsystem 660 at the rear of the helmet, the augmented reality display panel and driving front circuitry 672, the camera assembly 630, and the battery within the wings. Since the augmented reality display panel and driving front circuitry 672 are constrained to be located near the front of the helmet, the location of the integrated electronics subsystem 660 at the rear of the helmet provides a first balancing element. However, the integrated electronics subsystem 660 (e.g., including rear circuitry 668 and a heat sink 674, along with a corresponding fan and housing) is generally heavier than the display system located at the front. By carefully selecting the position of the wings 640, and thus extending the weight of the extended battery, the center of gravity 642 can be shifted forward and backward along the length of the helmet. Through user testing and force calculations, it has been found that in the design shown, positioning the wing 640 (via positioning the fixed housing 654 on the inner portion 632) to the rear of the safety goggle 620 positions the center of gravity 642 exactly at the rear of the wing 640 and below the edge or brim of the helmet. Because the helmet is constructed with a substantially symmetrical design, the right and left sides of the helmet are similarly (e.g., symmetrically) weighted, and therefore the center of gravity 642 of the helmet is located on or near the centerline of the helmet (e.g., along...). Figure 2G (Cross-section line A-A' in the diagram). For user comfort, it is preferable that the center of gravity 642 is located as low as possible above the user's head. By laterally mounting multiple batteries in the wing 640 and having a portion of the batteries reside within the wing 640 below the edge or brim of the helmet, the center of gravity 642 is lowered to below the edge or brim, improving comfort during extended wear.

[0124] Therefore, in Figure 6In the examples (and others shown herein), the placement of the batteries has been specifically configured for an optimized weight distribution in the helmet. Since batteries are typically the heaviest (or most significant) component in augmented reality devices, controlling their placement within the head-mounted device or helmet can increase user comfort and reduce and / or prevent neck strain. Prototyping revealed that a low center of gravity (e.g., below the edge or brim of the helmet) is most comfortable for extended use; therefore, the layout of the battery connection interface was carefully designed (e.g., as shown in the figure) to provide optimal comfort and minimal perceived weight. With a lower center of gravity of 642, as shown in the figure (achieved through proper arrangement of helmet components), users found a reduced perceived weight of the device and were able to move around on construction sites without pain. While this placement has been described with respect to helmets and offers particular benefit for construction sites involving extended use (e.g., site visits and inspections) and sometimes requiring physical access, compared to use in the home or office, it could also be used for augmented reality devices or other electronic head-mounted devices that do not include a helmet.

[0125] Exemplary bracket adjustment mechanism

[0126] Figures 7A to 7C An example bracket height adjustment mechanism is shown that can be used with the helmets described in the previous examples. The bracket height adjustment mechanism allows for easy and quick adjustment of the helmet's height on the user's head. Although designed for use with helmets including integrated electronic subsystems according to the foregoing examples, the bracket height adjustment mechanism can also be used with other forms of helmets and helms, including those without integrated electronic subsystems.

[0127] Figure 7A View 702 shows an example bracket height adjustment mechanism 705 used within an exemplary safety helmet 710. The bracket height adjustment mechanism 705 includes a bracket 755 for positioning the safety helmet 710 on a user's head and a set of bracket mounting pins 771. Figure 7A In the example, the brace 755 extends around the front and sides of the user's head and complements the rear adjustable neck brace 770 (e.g., as referenced). Figure 2C (As described in 270). In other examples, the bracket 755 may extend completely around the user's head and include a front, sides, and a rear. Figure 7A In the middle, the bracket includes a front part 755-A, a right side part 755-B, and a left side part 755-C (in Figure 7B and Figure 7C (As shown in the diagram). The bracket 755 may comprise a semi-flexible polymer. Figure 7AThe front and sides include areas 753 and 754 for attaching a comfort pad to the user's head. For example, areas 753 and 754 may include half of a hook-and-fastener system that allows easy attachment and removal of the comfort pad. In other examples, the comfort pad may be attached to areas 753 and 754 with adhesive. In use, a bracket 755 with the comfort pad, a rear adjustable neck brace 770, and a woven support net 748 can contact the user's head to allow the user to comfortably wear the helmet 710. (See at least the references...) Figure 3C As described, the braided support net 748 can be hooked onto the fastening tip 762 located inside the helmet 710 (e.g., inside a double-layer design).

[0128] Figure 7B A detailed view 704 shows a plurality of spaced-apart holes 764, 765 disposed within the bracket 755. These spaced-apart holes 764, 765 are adjustablely aligned with corresponding holes within a bracket mounting base that receives the bracket 755. The bracket mounting base is located in... Figure 7C This is shown in more detail below. For example... Figure 7B As shown, the set of bracket mounting pins 771 is removable to select different holes among a plurality of spaced-apart holes 764, 765 to adjust the relative height of the bracket relative to the bracket mount during use. For example, the height of the bracket 755 can be adjusted vertically in increments of 10 mm within the bracket mount (and via the extended safety helmet 710). In the example shown, the set of bracket mounting pins 771 includes a quarter-turn bayonet locking pin. The shaft 772 of each pin includes two laterally spaced lugs 773 near the base of the shaft. These lugs 773 include a downwardly sloping projection toward the base of the shaft 772. Each spaced-apart hole 764, 765 includes a central circular portion and two laterally spaced side recesses corresponding to the lugs 773. Figure 7C The bracket mounting areas 775 are shown in more detail, located within the helmet 710 in a position corresponding to the positions of spaced-apart holes 764, 765 on the bracket 755. The bracket mounting areas 775 collectively form a bracket mount. In this example, there are four bracket mounting areas 775 located near the four corners of the helmet 710 and corresponding to the four corners of the bracket 755. Each bracket mounting area 775 includes a hole 781 shaped to mate with the spaced-apart holes 764, 765 of the bracket 755 (e.g., having a central circular portion and laterally spaced notches).

[0129] To insert bracket 755, the user from Figure 7CThe configuration shown begins. Four bracket mounting pins 771 are provided, corresponding to four sets of spaced-apart holes 764, 765 and corresponding holes 781 in the bracket mounting area 775. The user aligns one of the spaced-apart holes 764, 765 with the mounting hole 781 to "select" a specific spaced-apart hole. The user then aligns the lug on each bracket mounting pin 771 with the notch in the hole 764, 765, 781 and inserts the pin 771 through the selected spaced-apart hole into the mounting hole 781. The user then rotates each pin 771 a quarter turn (e.g., 90 degrees clockwise or counterclockwise depending on the lug configuration) to rotate the lug away from the notch, thereby securing the lug to the back of the wall of the circular portion adjacent to the mounting hole 781. In this way, the bracket 755 is locked in place by a set of four bracket mounting pins 771 for the selected alignment. It should be noted that although four sets of pins, spaced holes, and mounting holes are described, different numbers of sets may be provided in other examples.

[0130] If possible Figure 7A , Figure 7B and Figure 7C As seen in each of the bracket mounting pins 771, each bracket mounting pin 771 may include a foldable handle 774 that, when at rest, is retracted around the circumference of the pin head. However, during adjustment of the bracket position, the foldable handle 774 can pivot away from the face of the pin head (i.e., pivot to a position substantially perpendicular to the face of each mounting pin) to facilitate rotation of the pin. The foldable handle 774 can then return to its flat (i.e., in line with the pin face) resting position so that the foldable handle 774 does not dig into the user's head when the helmet 710 is worn. The foldable handle 774 may include a bent metal wire or a molded polymer component that engages with a small hole on the circumference of the pin face. The foldable handle 774 may be designed to withstand frictional forces when at rest, which prevent it from rotating away from the face of the mounting pin unless moved by the user.

[0131] like Figures 7A to 7C As shown, a method is provided for adjusting the height of a safety helmet when positioned on a user's head. This method, for example, involves... Figure 7A The configuration shown begins. The method includes a first step: rotating a set of bracket mounting pins 771 to remove the pins from a set of corresponding holes 764, 765 in the bracket 755 and bracket mounting seat 775 of the helmet 710. This allows the user to remove the bracket, as... Figure 7CAs shown. Next, the user selects a set of alternating mounting holes in at least one of the bracket 755 and the bracket mount 775. In this example, the bracket 755 has two vertically spaced holes, and the bracket mount 775 has a single hole 781; however, in other examples, the bracket mount 775 may also have multiple spaced holes, or instead of multiple spaced holes within the bracket 755 (e.g., the bracket may instead have a single hole). In other examples, more than two different spaced positions can be provided to provide more than two adjustable positions. It should be noted that the user can choose to mix and match the hole pairs on the bracket and the bracket mount to provide different height configurations.

[0132] Returning to the method, the user moves at least one of the bracket and bracket mount to align with a selected set of alternating mounting holes. For example, in Figure 7A In the diagram, the rear pin 771 is shown in the upper bracket hole 764, and the front pin 771 is shown in the lower bracket hole 775. The user can remove one or more of the pins 771 to select different bracket holes. It should be noted that the user does not need to remove all the pins and can choose to adjust one (or a subset) of the bracket mounting pins at any given time. Once a selection is made and the desired hole is aligned, the user reinserts the bracket mounting pin 771 into the aligned alternative mounting hole (e.g., following a similar pattern). Figure 7B (The process is shown in the diagram). Finally, the user rotates one or more of the bracket mounting pins in the group to lock the pins in place (e.g., returning to a position similar to...). Figure 7A The configuration shown is a configuration with adjusted positions.

[0133] Figures 7A to 7C The example is characterized by the use of a quarter-turn pin to provide bracket height adjustment in a construction helmet. This example is an improvement on the contrasting "click-fit" adjustment mechanism (e.g., where the bracket has a row of holes on corresponding tips that snap into the helmet). The contrasting "click-fit" adjustment mechanisms are often cumbersome—disengaging the robust bracket connection is typically laborious and difficult, resulting in a trade-off between the robustness of the bracket installation and the ease of adjustment. In these contrasting cases, disengaging the bracket is often difficult, making it challenging for the user to operate the very robust bracket snap-fit ​​required for a secure connection. In contrast, this example provides a robust bracket connection (e.g., a locking mechanism via a bracket mounting pin) but also allows for easy access and adjustment (e.g., via a quick quarter turn of the pin, which can be facilitated by a foldable handle).

[0134] Exemplary handheld controller

[0135] Figures 8A to 8DA handheld controller 805 is shown that can be used with the augmented reality helmet described in the previous examples. The handheld controller 805 assists the user in interacting with both the visible real and virtual environments and has functions for configuring the augmented reality display. For example, the handheld controller 805 may include an improved version of the handheld controller described in WO2019 / 048866 A1. Therefore, the handheld controller can be able to implement the functions described in WO2019 / 048866 A1, but with additional new features not found in WO2019 / 048866 A1.

[0136] Figure 8A A top perspective view 800 of the front portion of a handheld controller 805 is shown. The handheld controller 805 includes a central body 810 configured to be gripped by a user's hand when viewing augmented reality images, for example, via a helmet as described in one of the aforementioned examples. Similar to the helmet of the previous example, the handheld controller 805 includes photoelectric sensors 812 distributed on the controller. These photoelectric sensors 812 are similar to those arranged on... Figure 2A The safety helmet 210 contains photoelectric sensors 212. These sensors enable the position and orientation (i.e., attitude) of the handheld controller 805 to be located within a tracking volume formed by a set of tracking beacons (such as those implementing tracking beacon 102, e.g., according to the safety helmet). Preferably, the position tracking system has millimeter-level accuracy. The handheld controller 805 includes an wing-shaped design characterized by right wing 814-A and left wing 814-B. Unlike the wing-shaped design of a single elongated body in WO2019 / 048866 A1, this wing-shaped design has several advantages. First, it protects the user's hand from impacts around the construction site, such as from objects moving toward the user's hand or when the user gestures with the handheld controller to control the augmented reality display. Second, the wing-shaped design allows for improved arrangement of the photoelectric sensors 212—by having a unique shape in which the photoelectric sensors 212 are more evenly distributed within the volume surrounding the central body 810, the determination of the attitude of the handheld controller 805 can be improved, for example, increasing accuracy, particularly for gesture movements. The front of the handheld controller 805, located on the top of the central body 810, includes a series of buttons and indicators, including a control pad 820. In this example, the control pad 820 includes four directional buttons and a central selection button. The control pad 820 may also include light-emitting diode (LED) indicators to show the status of the handheld controller (e.g., battery replacement, device power on, and / or device tracking). In other examples, the control pad 820 may include a (capacitive) touch pad or other user interface technologies.

[0137] At the front of the handheld controller 805 is a trident nose 830. The trident nose 830 includes an upper single fork and two lower forks (these are located at...). Figure 8D (See image). To configure the transformation between the coordinate system used by the location tracking system and the coordinate system used by the Building Information Model (BIM), the handheld controller 805 can be used to measure the positions of a series of control points. These control points may include adhesive targets, which are positioned within the construction site and then measured to determine the geodetic location of the target's center. The user operating the handheld controller 805 positions the two lower prongs at defined points on the target (e.g., lower corners or marked locations) and aligns the upper prong with the center of the target. The user can then activate a button on the handheld controller 805 (e.g., ...). Figure 8C and Figure 8D The control pad 820 shown has a central selection button or trigger button 816 to determine the position and orientation of the handheld controller 805 in the position tracking coordinate system. Since the tip of the fork has a known position relative to the defined origin or center / reference coordinates of the handheld controller, the accurate position and / or orientation of the target's center in the position tracking coordinate system can be obtained. Once this operation has been performed for three or four control points, two sets of coordinates for the control points—one set in the coordinate system used by the BIM (e.g., a geodetic coordinate system) and one set in the coordinate system used by the position tracking system—can be processed to determine the transformation mapping between the two coordinate systems (where a forward transformation maps one direction between the coordinate systems, and an inverse transformation maps the other direction). The transformation can be defined as a 4×4 matrix with rotation and translation terms. Further details of this calibration procedure are covered in WO2019 / 048866 A1.

[0138] Figure 8B A rear perspective view 820 of the top of the handheld controller 805 is shown. Besides... Figure 8A In addition to the features shown, a battery access door 840 is also shown. The handheld controller 805 can be used as... Figure 5B (550) or Figure 9C The rechargeable battery shown in (950) is a common battery type that can be used in any device described herein. The battery can be replaced by pressing the release button 841, for example, replacing a battery that has just been charged.

[0139] Figure 8CA side view 804 of the handheld controller 805 is shown. In this view 804, the central body 810 is visible. The user can hold the handheld controller 805 like a gun, with their palm and fingers wrapped around the grip portion 818. The grip portion 818 can be shaped to fit the contours of the human hand. The user then aligns their trigger (index finger) to press the trigger button 816. The trigger button 816 can be used to select and / or activate certain functions of the handheld controller 805 within an augmented reality interface visible via the augmented reality display panel (e.g., performing calibration measurements and / or activating references as described above). Figures 11 to 13 (Description of the direction of measurement).

[0140] Figure 8D A front view 806 of a handheld controller 805 is shown. In this example, the handheld controller 805 also includes an electronic rangefinder, which can be used in a set of user interface methods described later below. A portion of the electronic rangefinder is visible in front view 806, in addition to the components described above. In this case, the electronic rangefinder includes a laser ranging device. Therefore, the front of the handheld controller 805 includes a laser emitter 842 and a laser receiver 844. The laser emitter 842 emits electromagnetic radiation pulses that can be reflected from a surface and returned via the laser receiver 844. The laser receiver 844 includes a lens for focusing the returned radiation. The laser ranging device can be a laser rangefinder and can operate according to one or more of the following: time-of-flight measurement, multi-frequency phase-shift measurement, and interferometry. The laser rangefinder can include commercially available laser rangefinders. In other examples, the electronic rangefinder can use ultrasonic technology, optics (e.g., based on coincidence or stereo measurement), and / or an infrared rangefinder, etc. In one case, a button and / or trigger 816 on the control pad 820 can be used to activate the electronic distance measurement. The control pad 820 can also indicate to the user, for example, via LEDs or a panel display, whether an electronic distance measurement is in progress.

[0141] Figure 8D Electrical port 846 is also shown. This can include something similar to... Figure 2H and Figure 5A The USB-C port shown is port 268 or 568. Electrical port 846 can be used to power the handheld controller 805 and / or transmit data to and / or from the controller. For example, the handheld controller 805 can be plugged into a helmet and / or another computing device to download measurement data (including control point locations) acquired during use. In some configurations, the handheld controller 805 may also include a wireless communication interface (e.g., Bluetooth®, Zigbee®, and / or WiFi®) to transmit data.

[0142] Exemplary tracking beacon

[0143] Figures 9A to 9C This shows what can be used to implement Figure 1A and Figure 1B Different views of the exemplary tracking beacon 910 of the tracking beacon 102 shown. Figure 9A A front perspective view of 900 is shown. Figure 9B A first rear perspective view 902 is shown, and Figure 9C A second rear perspective view 904 is shown. The front of the tracking beacon 910 includes a window portion 912, behind which a transmitter for emitting one or more of beams 103, 104, and 105 is mounted. The tracking beacon also includes an electrical port 913, which may be similar to the one referenced above. Figure 2H , Figure 5A , Figure 8D The described ports are USB-C ports 268, 568, and 846. Electrical port 913 can be used for one or more of the following: power, data communication, configuration, and firmware updates. The rear of the tracking beacon 910 includes a battery access button 914 and a battery access door 918. (As...) Figure 9C As shown, when the battery access button 914 is pressed, the battery access door 918 opens (e.g., by releasing the spring that pushes the door open).

[0144] The tracking beacon 910 in this example is configured to use a removable, rechargeable battery 950. The removable, rechargeable battery 950 can be of a type usable by all the devices described herein (e.g., helmets and handheld controllers). As mentioned above, this simplifies charging and battery replacement. Figure 9C In this configuration, a newly charged battery 950 is inserted into the interior 922 of the tracking beacon 910, causing the terminals 952 of the battery 950 to make electrical contact with the corresponding battery interface within the tracking beacon 910. The user closes the battery access door 918 by rotating it about the base pivot axis, overcoming a spring-loaded force. A leaf spring-type actuating member 924 is located on the underside of the battery access door 918. Similar to... Figure 5G The battery biasing member 590 pushes the member 924 to apply force to the base of the battery 950, so that the battery terminal 952 forms a firm connection with the corresponding battery interface in the tracking beacon 910.

[0145] Exemplary charging station

[0146] Figure 10A and Figure 10B This is a perspective view showing two configurations 1000 and 1002 of an example charging station 1010. The example charging station 1010 can be used to charge multiple rechargeable batteries. The rechargeable batteries can have, for example... Figure 5B and Figure 9CThe design shown. Rechargeable batteries can include a common (i.e., shared) battery for a kit of interconnected augmented reality components used on a construction site. For example, a safety helmet can use two rechargeable batteries (e.g., such as...). Figure 5A As shown), the handheld controller can use a rechargeable battery (e.g., as shown). Figure 8B (as shown), and the tracking beacon can use a rechargeable battery (e.g., as shown). Figure 9C (As shown). Rechargeable batteries may include lithium-ion batteries.

[0147] exist Figure 10A The image shows a first configuration 1000 of charging station 1010. Charging station 1010 includes multiple (eight in this example) charging bays 1020 for receiving rechargeable batteries for recharging. Charging station 1010 can be plugged into a power source via power port 1012. Charging station 1010 can provide a fast charging mode where one or more rechargeable batteries are rapidly charged using a high current. In one case, charging station 1010 can have a power output exceeding 100W for charging. Powering via a USB port allows charging modes with one of 5V / 3A, 9V / 3A, 15V / 2A, and 20V / 1.5A, and the battery port can allow charging at approximately 8.4V / 1.19A. Voltage and current supply can be configured in the firmware for charging station 1010. The charging rate for each charging mode can also be configurable, for example, depending on how many batteries are connected to charging station 1010. Assuming 90% efficiency and taking into account a built-in fan for cooling, the maximum power consumption of the charging station is likely around 158W. The charging process may have two or more phases: a constant current phase, in which the voltage increases toward its peak; a saturation phase, in which the voltage reaches its peak and the current then decreases (the voltage may remain constant at its peak); and a trickle or replenishment phase when the battery is fully charged (e.g., determined by a set percentage of the current reaching its initial constant charging current). Each charging compartment includes a battery connection interface 1022, which includes corresponding terminals on the rechargeable battery (e.g., ...). Figure 5B and Figure 5C The electrical terminals (552-A) are matched with the terminals in the battery charging station 1010. In this example, four battery charging compartments 1020 are connected in series on each side of the battery charging station 1010. This facilitates efficient layout in construction areas with limited space.

[0148] In this example, charging station 1010 includes a pivoting base portion 1030. In use, the base portion 1030 pivots outwards, exposing the battery connection interface 1022 in each battery charging compartment 1020 and enabling it to receive batteries for charging. However, when one side of the battery charging station 1010 is not in use (e.g., when the charging station is transported to and / or from a construction site), the base portion 1030 can pivot upwards, as... Figure 10B The second configuration 1002 is shown. As can be seen, in this second configuration 1002, the battery charging compartment 1020 is closed, and the battery connection interface 1022 is protected, for example, to prevent dust and dirt from entering the interface. In one case, the base portion 1030 on each side can pivot independently. In another case, the base portions 1030 on both sides can be driven to a common gear mechanism that coordinates the opening of the charging compartment. In yet another case, the base portions 1030 can pivot individually for each compartment 1020. In an alternative example, the pivoting base portion 1030 can also be replaced by a sliding base portion 1030 that can be translated into and out of the body of the charging station 1010 to expose or conceal the battery connection interface 1022. Similarly, each base portion 1030 can be translated as a side unit or used individually for different compartments (e.g., depending on the chosen implementation design).

[0149] Therefore, in some cases, Figures 2A to 10B The components provide a kit for use on construction sites. For example, the kit may include one or more of the following: a helmet with an integrated augmented reality subsystem; multiple removable rechargeable batteries; a set of removable battery housings, each receiving one of the multiple removable rechargeable batteries in use, at least two of the removable battery housings being mechanically coupled to the helmet in use to power the helmet's integrated augmented reality subsystem; and one or more tracking beacons for determining the helmet's location on the construction site, each tracking beacon configured to receive at least one of the multiple removable rechargeable batteries for power generation when no external power source is available. The kit may also include, for example, Figures 8A to 8D One or more of the handheld controllers and charging stations shown are used to recharge one or more of a plurality of removable rechargeable batteries, such as... Figures 10A to 10B As shown.

[0150] The charging station can be arranged to simultaneously recharge more than two of a plurality of removable rechargeable batteries, and can include a plurality of battery recharging compartments on each side of the charging station. The receiving portion (e.g., base portion 1030) on each side of the charging station can move between two positions: an open position for receiving one or more of the plurality of removable rechargeable batteries and a closed position for protecting the terminals of the plurality of battery recharging compartments (e.g., as shown in the first configuration 1000 and the second configuration 1002, respectively).

[0151] Exemplary User Interaction Methods

[0152] Figure 11 Example 1100 illustrates the use of a handheld controller as a mobile device to interact with an augmented reality view of a construction site. Figure 11 In the example, the helmet 1110 with an augmented reality display is used to view augmented reality images. The helmet 1110 may include the helmet of the previous example (e.g., as referenced). Figures 2A to 7C (As described) or may include another device. The handheld controller 1120 may include Figures 8A to 8D The handheld controller 805 or another mobile device. The handheld controller 1120 can be used to interact with a virtual representation of the construction site viewed by a user with a head-mounted display (e.g., a user wearing a hard hat 1110). In this case, the handheld controller 1120 is detached from the hard hat 1110. The handheld controller 1120 includes a set of sensors for a position tracking system (e.g., ...). Figures 8A to 8D The photoelectric sensor 812) and electronic rangefinder (e.g., Figure 8D A laser measuring device having a transmitter 842 and a receiver 844. This set of sensors is configured to acquire sensor data to derive one or more of the position and orientation of the handheld controller 1120 within the construction site. Preferably, this is relative to a position tracking system (e.g., Figure 1A The system shown determines both the position and orientation (i.e., attitude) of the handheld controller 1120. For example, the center point or origin of the tracking controller can be defined within the coordinate system of the position tracking system (e.g., where the origin of the position tracking system can be defined relative to the tracking beacon 102—in some cases, one of the tracking beacons represents the origin or zero point, or a corner of the tracking volume is set as the origin or zero point). The position tracking system can have millimeter-level accuracy.

[0153] In use, the electronic rangefinder is configured to determine the distance from a known location on the handheld controller 1120, along a line of sight, to an occupied portion of the space within the construction site. This is in Figure 11As shown in the diagram. In this example, a laser beam is emitted from the laser emitter 842 along line of sight 1122. It then encounters a point 1132 in the occupied portion of the space (in this example, a point on the plane of wall 1130) and is reflected back to the handheld controller 1120 along line of sight 1124, where it is received by the laser receiver 844. This process can occur, for example, when a user points with the handheld controller 1120 and presses the trigger button 816 in a given operating mode. Figure 8B As shown. In the current case, the occupied portion of the space is away from the handheld controller 1120, for example, it includes a wall 1030 located at a distance from the user. By using known techniques for electronic distance measurement (e.g., reference...) Figures 8A to 8D One of the methods described can measure the distance 1126 between the handheld controller 1120 and the wall 1130 along the line of sight 1122, 1124. Using sensor data and the determined distance, the position of a point 1132 corresponding to an occupied portion of the space can be determined, the position being defined with reference to the position tracking system. Therefore, the handheld controller 1120 is configured to be oriented by the user within the construction site to compare the point defined in the virtual representation with the measured real-world point. For example, the position of the real-world point 1132 in the coordinate system of the position tracking system can be determined based on the orientation of the handheld controller 1120 measured in the coordinate system of the position tracking system and the distance measured by an electronic rangefinder. This then positions the point 1132 in the virtual space. Thus, the point 1132 can be shown in the augmented reality view within the helmet 1110. For example, the user can place virtual objects 1140 at the point location, such as against the wall 1130. Alternatively, they can associate virtual annotations (such as virtual “sticky notes”) with the point 1132. This then provides a method for annotating virtual equivalents of real-world points (such as point 1132 on a wall) in a virtual model (such as BIM). For example, during an inspection, a user might want to annotate a digital version of a real-world object. Therefore, the user points controller 1120 at the object and clicks a trigger button (or activates another input mechanism), and they are able to determine the virtual equivalent point within a virtual coordinate system. Points determined in the location coordinate system can also be mapped to points in the BIM via a calibration transformation between the location coordinate system and the coordinate system used by the BIM. Thus, annotation of the BIM is possible.

[0154] More specifically, in a preferred embodiment, the electronic rangefinder emits a directional beam to determine the distance. As previously described, the directional beam may include an electromagnetic radiation beam or an ultrasonic beam. The directional beam is emitted from a known location on the mobile device, such as transmitter 842 on the handheld controller 805. The known location has a fixed position relative to a tracking reference point of the mobile device. For example, the tracking reference point of the handheld controller 805 may include an inertial measurement unit (IMU) within the body 810 of the handheld controller 805. The IMU has a fixed position relative to transmitter 842, for example, a fixed transformation (translation and rotation) between the IMU position and transmitter 842 that can be derived from the computer-aided design (CAD) specifications of the handheld controller 805. Therefore, if the attitude of the mobile device is known (e.g., the position of the IMU within the device and the orientation of the device at that position, represented by a normal vector), the position and orientation (i.e., attitude) of transmitter 842 can be known using a fixed transformation and a defined line-of-sight transformation that models a line from the emission point with a length equal to the measurement distance. This then allows the attitude of measurement point 1132 to become known. In practice, calibration can be performed to set the line-of-sight transformation (e.g., it can be considered as a projection from transmitter 842 or at the midpoint between transmitter 842 and receiver 844) – the precise line-of-sight transformation can depend on the configuration of the electronic rangefinder (e.g., the effective or modeled beam projection of the transmitter 842 and receiver 844 components can be set in the manufacturer's datasheet). In practice, there exists a known or measurable emission vector originating from a known location on the handheld controller 1120, and this point can be determined using the projected line and the measured distance. In parallel, a similar process can also be performed in a virtual space used for modeling BIM and / or other information, and thus the directional beam can also be visually represented in the virtual space (e.g., as a cylinder with a set diameter projected from a model of the handheld controller based on the controller's set dimensions). In one case, the CAD model of the handheld controller can be used to create a virtual model of the controller (e.g., using dimensions defined within the CAD model). Therefore, the transmission vector and the determined distance can be used to determine the three-dimensional position of the point corresponding to the occupied portion of the space relative to a known position on the handheld controller (e.g., the transmitter 842 position), and the known position is a known or measurable position in three-dimensional space relative to the position of the mobile device derived from the sensor data (e.g., it can be derived from a fixed transformation that associates the transmitter 842 position with the IMU position that forms the reference point or origin of the handheld controller 1120).

[0155] Figure 11 Therefore, an example of a method is shown, which can be used to implement a human-computer interface for augmented reality applications. Figure 12A The method is described in more detail in flowchart 1200.

[0156] exist Figure 12A At step 1212, the method includes tracking the location and orientation of mobile devices within the construction site. This may include using... Figures 8A to 8D The photoelectric sensor 812 and Figure 1A The position tracking system shown tracks the attitude of the handheld controller 1120. As described above, the attitude of the handheld controller can include a six-DOF vector specifying the position of a known reference point on the handheld controller within the coordinate system of the position tracking system (e.g., a defined IMU position) and its orientation relative to the axes of the coordinate system. The attitude can be defined as described earlier in this document. In step 1214, the user points to a first point with the mobile device. In the case where the mobile device includes the handheld controller 1120, this can include pointing the handheld controller 1120 at an object of interest. The object of interest can be in the real world (e.g., a point on a wall 1130) or a virtual object visible to the user via an augmented reality display such as a head-mounted display provided by the helmet 1110. In the former case, the handheld controller 1120 can include a laser pointer and emit a laser beam with a visible point (this can be a laser beam for directional distance measurement or a separate visible beam, such as an infrared laser beam for distance measurement, to supplement an invisible ranging beam). Visible points can be displayed on real-world objects (e.g., as red dots), and when a visible point coincides with an object the user wants to interact with, the user can input data via a mobile device's user input device (such as...). Figure 8C The trigger button 816 in the display indicates this. Therefore, the mobile device may include a laser pointer and an electronic rangefinder, or the laser pointer may be part of the electronic rangefinder. If the user is viewing a virtual object, the augmented reality display may show a virtual line projected from the virtual model of the mobile device, which stops when the virtual line intersects with another virtual object in the virtual space (e.g., the user can point to it). Figure 11 A virtual representation of window 1140 in the wall 1130, even if this is not yet constructed within the wall 1130. The method described herein is flexible and can be used in either context (e.g., mapping from real-world points to virtual points and vice versa).

[0157] At step 1216, a directional ranging beam is emitted from the mobile device in the direction of the indicated first point. For example, in Figure 11In the diagram, the indicated point is 1132, and the directional ranging beam is shown as 1122. When the trigger button 816 is pressed, the directional ranging beam can be emitted. In step 1218, the directional ranging beam is used to determine the distance to an occupied portion of the space within the construction site. For example, when using a directional ranging device, a laser or ultrasonic beam can be emitted, which travels through an open space until it encounters a real-world surface, whereby the surface reflects the beam toward the mobile device (e.g., as shown in the image). Figure 11 (As shown in 1124). The reflected beam is then received by a mobile device (e.g., in...). Figure 8D The distance is determined using a known method of comparing the transmitted and received directional ranging beams (e.g., using time of flight and / or phase difference). If the user points to a virtual object, such as that shown on an augmented reality display (e.g., a helmet 1110), the transmitted directional ranging beam will not be reflected by the virtual object, but only by the corresponding real-world surface. This can be used during inspection; for example, if the user points to a pipe fitting in virtual space, the virtual distance can be determined by ray tracing from the modeled handheld controller to the virtual object within the virtual space. This can then be compared with the actual measured distance determined by reflection from the corresponding real-world object (e.g., a pipe fitting installed on a construction site) by the directional ranging beam. A mismatch between the virtual distance and the distance measured in the real world indicates a difference between the virtual model (e.g., BIM) and the constructed environment.

[0158] In step 1220, the direction of the directional ranging beam is determined. As described above, this may include using the attitude of the handheld controller 1120 and a fixed transformation (e.g., a 4×4 matrix) derived from the controller's CAD specifications to determine the launch position (i.e., the modeled launch position, where small differences may exist between actual launch positions depending on the configuration of the ranging device) and launch orientation. For example, the attitude of the modeled beam launch position on the handheld controller 1120 can be calculated (e.g., by applying a fixed transformation to a tracking reference point on the handheld controller 1120). Therefore, at step 1222, the position and orientation of the mobile device, along with the direction of the ranging beam and the distance to the occupied portion of space, can be used to determine the position of the second point corresponding to the first point.

[0159] If the first point is a real-world point (e.g., point 1132 on physical wall 1130), a virtual point corresponding to the real-world point can be determined in the virtual world. This can be achieved using a model of the mobile device in the virtual world (e.g., the coordinate system of a position tracking system). The tracked pose of the mobile device in the virtual world is known via the position tracking system, and the pose of the modeled launch position is also known according to step 1220. In the virtual world, a ray can be traced from the modeled launch position, for example, along the normal to the surface of the modeled launcher 842, and at step 1218, it continues at a distance equal to the measured distance. Thus, a point at the end of the ray at the measured distance is determined, which is a virtual point (e.g., a point in a virtual coordinate system, such as the position tracking system coordinate system) corresponding to the indicated real-world point. This virtual point can be further mapped between virtual coordinate systems, for example, it can be mapped to the BIM coordinate system using a calibration transformation that maps the position tracking system coordinate system to the BIM coordinate system. Thus, the BIM location of the real-world point can be determined.

[0160] If the first point is a point in the virtual world, the corresponding real-world point can be determined. For example, within an augmented reality display, a virtual line can be traced from the modeled launch position of the mobile device (e.g., similar to 1122 but only visible via the augmented reality display) until it intersects with an occupied portion of the virtual space (e.g., a surface or object within the virtual space—such as a BIM object that has been mapped to a position-tracking coordinate system for display). This provides the first point as the virtual point location. Using the calibration transformation described earlier, the virtual point location can be mapped to a corresponding point in a BIM coordinate system, which can be a real-world coordinate system (e.g., representing geodetic or geocentric coordinates). Now, in step 1222, the measured distance to the corresponding real-world point with the same pose of the mobile device is also known. Therefore, the position of an object or surface in the real world along the same tracked ray can also be determined. For example, the pose of the handheld controller 1120 in the BIM coordinate system represents the real-world position of the handheld controller 1120. This can be obtained using a calibration transformation. The measured distance can be added to this (e.g., via a fill transformation) to obtain the real-world position corresponding to the initially indicated virtual point. If a real-world surface or object is correctly modeled in the virtual world, mapping real-world points back to virtual space (e.g., from the BIM coordinate system to the location tracking coordinate system using a calibrated transformation) should provide the original indicated virtual point. However, if a mismatch exists between the virtual model and the real world, the second virtual point mapped from the real-world point may differ from the first indicated virtual point. This difference can be noticed (e.g., in BIM) because one or more of the model (e.g., BIM) and the real-world building may need adjustment. Therefore, this method can be used as part of a site inspection to check whether the virtual model of a planned building element matches its real-world building equivalent. Since users can point to any object in the real or virtual world, this provides a flexible and powerful way to quickly check whether a building conforms to design specifications.

[0161] As described above, in one scenario, data from Building Information Modeling (BIM) is used to populate the virtual space. BIM is defined relative to a model coordinate system, which can include a geodetic coordinate system or a geocentric coordinate system. In this case, the tracking at step 1212 is performed within the tracking coordinate system. This can include... Figure 1A The position tracking system shown may be a coordinate system used for different tracking systems (e.g., SLAM systems). The directional ranging beam is emitted from the mobile device and reflected by the occupied portion of space (e.g., as shown). Figure 11(As shown). The reflection of the directional ranging beam is detected by the mobile device. In these cases, the distance to the occupied portion of the space and the direction of the directional ranging beam are determined within the tracking coordinate system. For example, the direction of the directional ranging beam can be known by measurement of the modeled launch position or by a determined attitude (e.g., determined by tracking the device position plus a fixed transformation). The modeled launch position can be a predefined location on a handheld controller where the directional ranging beam is reflected. The measured distance can then be used to project a line having a length starting from the modeled launch position (e.g., when the beam is perpendicular to a fixed surface (such as the face of launcher 842) or projected onto a fixed surface (such as the face of launcher 842) at a set angle. Thus, the location of a real-world point within the construction site is determined within the tracking coordinate system (e.g., as a projected line of the modeled launch position plus the measured distance length).

[0162] In some examples, a calibrated transformation is used to determine the correspondence between the tracking coordinate system and the model coordinate system, mapping points between the coordinate systems. The calibrated transformation can be determined by measuring the location of survey control points with a handheld controller, thereby obtaining pairs of points (tracking coordinate system points, model coordinate system points), where multiple of these points can be compared to derive the calibrated transformation (e.g., via least squares and / or optimization).

[0163] Such as about Figure 12A The method discussed herein may include points on surfaces or objects defined as part of a building information model. In this case, the method includes mapping between a tracking coordinate system and a model coordinate system using a calibrated transformation to determine the corresponding positions of virtual and real-world points in a common coordinate system, and to identify any discrepancies between the indicated corresponding positions of virtual and real-world points in the common coordinate system. As will be understood, if forward and reverse calibrated transformations are obtained, comparisons can be made in either the tracking or model coordinate system (which may also be a real-world coordinate system), provided that the points are mapped to the coordinate system to be compared.

[0164] In one scenario, the method may include the difference between the corresponding positions of virtual points in a common coordinate system in a virtual space viewed by the user and real-world points. For example, these two points may be mapped to a tracking coordinate system and displayed in a 2D projection used to generate an augmented reality image for display. If the user points the handheld controller 1120 at a physical wall 1130 and measures a real-world point 1132 on the wall, the position of the real-world point 1132 in the real world can be determined by using the pose of the handheld controller 1120 mapped to a model (e.g., BIM) coordinate system and ray tracing of the measured distance 1126. If this point, represented in the model coordinate system, differs from the model of the wall, also represented in the model coordinate system (e.g., if the measured point is a few centimeters in front of or behind the modeled surface of the wall in the BIM), a mismatch (i.e., discrepancy) exists between the BIM and the construction site. If the user wishes to view the mismatch, both the real-world point and the modeled wall in the model coordinate system can be mapped to the tracking coordinate system and displayed within the augmented reality image. Similarly, if the user points to a model of a wall as seen in the augmented reality image, ray tracing can be performed according to... Figure 12A The method determines virtual points on the model of the wall in the tracking coordinate system. The virtual points corresponding to the measured distance 1126 can also be shown, and the two can be compared in the tracking coordinate system, where any differences can be highlighted in the augmented reality interface.

[0165] In some cases, if such a discrepancy is perceived between the model and the measured real world, the user can, for example, instruct the model to be updated to reflect the measured reality via an augmented reality user interface. This can include receiving instructions from the user to match virtual points with real-world points in a common coordinate system; for example, receiving instructions to match points on the model representing walls in the common coordinate system with the measured real-world point locations. In some cases, "matching" points can include updating the location of surfaces or objects within the BIM.

[0166] In such Figure 11 In some cases, the mobile device includes a handheld portable construction tool. In these cases, the instruction in step 1214 may include pointing a virtual representation of the handheld portable construction tool to a virtual point of interest, performing ray tracing from a predefined location (e.g., a modeled emission location) on the virtual representation of the handheld portable construction tool to a virtual surface or object in virtual space, and determining the location where the ray from the ray tracing intersects the virtual surface or object, said location being presented as the location of the indicated virtual point. This indication can then be compared to a measured real-world point.

[0167] In an alternative example, the mobile device may be worn by the user and include a head-mounted display. For example, the mobile device may include a helmet 1110. In this alternative example, the instruction in step 1214 may include pointing virtual representations of one or more body parts of the user to a point of interest. For example, using hand tracking and / or other gesture recognition libraries, such as... Figure 2A The illustrated camera assembly 230 can be used to determine the position of one or more of a user's arm, hand, and fingers within a tracking coordinate system. In this case, the user can point to a point of interest as an alternative to using the handheld controller 1120. This method can be used with or without the handheld controller. However, the method for determining and comparing points can be similar. For example, the method can include ray tracing from a position defined by a virtual representation relative to one or more of the user's body parts to a virtual surface in virtual space. This could involve determining the position of the axis of the finger the user is pointing at and ray tracing along that axis a line starting from the fingertip. Similarly, the location where the ray-traced light intersects the virtual surface can be determined, and this location is presented as the position of the indicated virtual point. In this case, an electronic rangefinder can be mounted within the camera assembly 230 and used to determine the corresponding real-world distance measurement. This can be used if the user points at a specific virtual plane with their finger and the distance to the corresponding real-world plane is measured from the head-mounted display. In this case, the indication of the virtual point may not require a precise pointing device, as the ray can be traced and the first object intersecting the ray can be determined in virtual space. In this scenario, the position tracking system could include a SLAM system that uses image data from camera component 230 to determine the distance to a real-world surface (e.g., a real wall on a construction site). In this case, the model of the wall and the actual measured location of the wall can be compared again.

[0168] In some cases, the position and / or orientation of one or more of the handheld controller and the modeled launch position can be determined as a statistical measure of multiple measurements. For example, the position of the handheld controller can be sampled to determine an average position value. This can also be performed using distance measurements; for example, multiple distance measurements can be averaged over a short time period (e.g., milliseconds) and the average distance can be used in future calculations. Furthermore, short-term tracking information from one or more IMUs can be used to determine one or more of the position and / or orientation (e.g., via fused output measurements). These methods can contribute to accuracy and reduce anomalous measurements, such as those caused by measurement errors or human factors. For example, a user's hand can move a certain amount within 100-200 milliseconds, so calculating and using averaged pose and distance can improve accuracy.

[0169] In some cases, the directional ranging beam is emitted from a defined location on the mobile device, and the direction of the directional ranging beam is determined based on the orientation of the mobile device. For example, the defined location may be set based on design measurements of the device (e.g., CAD drawings). If the transmitter is arranged parallel to the axis of the mobile device (and therefore the directional ranging beam is emitted parallel to that axis), then that direction may be the orientation of that axis. For example, the axis may include the top surface of the handheld controller 805 and... Figure 8A The upper single fork shown is aligned with the axis.

[0170] In some cases, directional ranging beams can be emitted from a defined location on a mobile device with configurable directionality. These cases can differ from, for example... Figures 8A to 8D The handheld controller 805 is shown. For example, an electronic ranging device can have a movable transmitter-receiver assembly (with...). Figure 8D (As shown in the transmitter-receiver assembly fixed relative to the handheld controller body). In these cases of movable transmitter-receiver assemblies, determining the direction of the directional ranging beam includes measuring configurable directivity during transmission. For example, the movable transmitter-receiver assembly may be mounted within a gimbal that includes electronic sensors for measuring the orientation of the assembly within the gimbal.

[0171] In some cases, the position and orientation of the mobile device are provided as a six-DOF pose within the tracking coordinate system. In these cases, the distance to the occupied portion of space and the direction of the directional ranging beam are used to determine the transformation within the tracking coordinate system, which defines the position of the real-world point within the tracking coordinate system. For example, this could be a transformation relative to a reference point or origin of the mobile device, where the rotation term is based on the orientation of the mobile device, and the translation term is based on the position of the mobile device and the measured distance.

[0172] In cases where the first point includes a virtual point, the method may include using the location and orientation of the mobile device, as well as the direction of the ranging beam and the distance to the occupied portion of the space, to determine the location of the corresponding real-world point of the virtual point. For example, if a user indicates a virtual object such as a wall or column in a model, the real-world measured distance along the ray representing the emitted directional ranging beam to the tracked mobile device can be used to determine the location within the model coordinate system representing the real world. In this case, the method may further include mapping the real-world point back to the virtual space using a calibrated transformation between the coordinate system of the virtual space (e.g., the BIM coordinate system) and the coordinate system used for tracking in the real-world space (e.g., the tracking coordinate system). The locations of the mapped real-world point and the initially indicated virtual point in the virtual space can then be shown, including indicating any differences between the mapped real-world point and the virtual point.

[0173] If the first point includes a real-world point, the method may include indicating the first point by pointing a mobile device at the first point within the construction site. In this case, determining the location of the corresponding second point may include: determining the location of the first point in a coordinate system used to track the mobile device within the construction site; mapping the location of the first point to a virtual space to determine the location of the corresponding second point, which includes a virtual point within the virtual space; and indicating the location of the corresponding second point within the virtual space to the user via a head-mounted display.

[0174] In the examples described herein, computations for mapping between spaces and / or executing methods can be performed on a single device or on a distributed system of devices. In one case, the handheld controller 1120 may include a computing module (e.g., at least one processor and memory) to perform computations within the controller. In one case, the handheld controller 1120 is configured to wirelessly exchange data with the helmet 1110. In one case, a tracking computer server communicates with both the helmet 1110 and the handheld controller 1120 to determine their position and orientation based on sensor data. In other cases, tracking can be performed within the integrated electronic subsystem of the helmet 1110. Typically, depending on the system configuration, computations can be distributed among the devices. For example, for the independent use of a component suite including a tracking beacon, helmet, and handheld controller, computations can be performed within the integrated electronic subsystem 260, as referenced. Figures 2A to 2H (And others) as described. Although presented with reference to simple objects such as walls. Figure 11 This is just one example; however, real-world and modeled objects can take many forms (e.g., a user can point to any object in the real world and indicate any object defined in or loaded from BIM). For example, objects can include steel beams, columns, foundations, pipes, cables, HVAC (heating, ventilation, and air conditioning) units, windows, openings, doors, and more.

[0175] Figure 12B This is a flowchart 1250 illustrating another method in which a mobile device (such as a handheld controller as described herein) can be used to interact with an augmented reality system. Figure 12BThe method is an example approach to align virtual objects with real-world locations. In step 1262, virtual objects are obtained within the augmented reality user interface (e.g., as seen by the wearer of the hard hat 1110). This can include generating (i.e., producing) virtual objects from a defined list of virtual objects. For example, BIM may have multiple defined objects or assets representing components being built. Users can obtain virtual objects using a graphical user interface (GUI) displayed within the augmented reality view. Virtual objects may include pre-existing objects within a model of a proposed construction project (e.g., a model of a wall to be built) selected from a pre-existing list of objects, or new objects generated from a selected object template. In one case, virtual objects may not represent real-world objects but may instead include the form of virtual annotations.

[0176] In step 1264, the user selects a surface of the virtual object obtained in step 1262. For example, if the virtual object includes a cuboid object, a face of the cuboid object can be selected. A handheld controller 1120 can be used to perform the selection to point to a surface within the virtual space (e.g., using the ray tracing method discussed above). At step 1266, the user uses a mobile device to indicate a set of real-world points (e.g., using the reference above). Figure 11 and Figure 12A (Similar to one or more methods described in [the previous section]). Then, the positions of virtual points corresponding to multiple real-world points are determined. For example, when manipulating a virtual object within a tracking coordinate system, a virtual equivalent of the measured real-world point can be determined in the tracking coordinate system (using an electronic ranging device) (e.g., by calculating the position in the tracking coordinate system using acquired sensor data and / or mapping the calculated position from the model coordinate system to the tracking coordinate system using a calibrated transformation). If three or more points are indicated, a plane can be identified within the virtual space. In step 1268, the selected surface of the virtual object is then aligned with the measured real-world points, as represented within the virtual space (e.g., the tracking coordinate system). The aligned virtual object can be shown to the user on an augmented reality display. Effectively, the method generates a representation of a real-world surface using the measured points and then uses this representation to align the virtual object.

[0177] In some variations of this method, only one measured real-world point may be required. For example, a user can indicate a real-world point on a wall or other surface, and a reference virtual object representing that wall or surface can be identified in the BIM. If the measured real-world point matches the reference virtual object in the BIM (e.g., as it is then mapped to a tracking coordinate system), the selected face of the obtained virtual object can be mapped to the surface of the pre-existing reference virtual object. The reference virtual object and / or the plane generated from multiple measurement points may or may not be displayed. If not displayed, the obtained virtual object will appear aligned with a real-world surface or object when the user views the augmented reality image. Therefore, this method can be used to perform a "click-to-fit" alignment of a virtual object with its corresponding virtual and / or real-world object.

[0178] Figure 13 Showing the execution Figure 12B Example 1300 of the method, which is used with Figure 11 The diagram shows similar schematics and components. On the left side of the figure, the user moves the handheld controller 1120 to generate a virtual object 1310, and then selects a face 1312 of the virtual object. The dashed lines in this first step illustrate tracing light in the virtual world, but in the real world, electronic ranging devices are not used to emit ranging beams. Therefore, the left side of the figure reflects... Figure 12B Steps 1262 and 1264. Following this, moving to the right as indicated by the arrow in the diagram, the user then uses the handheld controller 1324 to indicate three points 1322 on surface 1320, which may include walls or partitions within a construction site. In this case, an electronic rangefinder, as can be seen through the rangefinder beam 1324, is used to determine the real-world distance of each point 1322 from the tracking position of the handheld controller 1120. The user can use the visible laser beam to indicate each point on surface 1320 and then tap the trigger button 816 (e.g., ...). Figure 8CAs shown, the distance to each point is measured. The set of three points is then mapped to a coordinate system used for displaying the virtual object 1310 within the augmented reality display on the helmet 1110; in this case, this coordinate system is the tracking coordinate system. If the virtual object is viewed outside the helmet 1110 (e.g., on a remote computer monitor or mobile device touchscreen), the coordinate system can be a model (e.g., BIM) coordinate system (because this doesn't need to be oriented with the view of the helmet 1110). In at least one of multiple coordinate systems, a plane 1326 comprising the three points 1322 is determined (e.g., via a known plane fitting algorithm). The selected plane 1312 is then aligned with the plane 1326 (e.g., if they are both defined and / or mapped to a common coordinate system, such as the tracking coordinate system used for display in the helmet 1110). Figure 13 The rightmost image shows an aligned virtual object 1330, where the selected face 1312 is configured to reside within plane 1326. Therefore, to the user, the virtual object 1330 appears to be "cropped" to the real-world surface 1320. In one case, in "alignment" mode, the virtual object 1310 can be automatically aligned with plane 1326 because a third point 1322 is indicated in the set. This method typically includes acquiring the virtual object within virtual space, using a movable device to indicate multiple real-world points, determining the position of the virtual point corresponding to the multiple real-world points, and aligning the virtual object within virtual space based on the position of the virtual point. Figure 13 As shown, in some cases, this can specifically include selecting the faces of a virtual object, using virtual points to define planes within the virtual space, and aligning the faces of a virtual object with planes in the virtual space.

[0179] In one scenario, a variation of the above method may include mapping a set of virtual points indicated on a virtual object in the augmented reality view to a set of real-world points measured by the handheld controller 1120. For example, a user may indicate a series of corners that form part of an object in the virtual space seen by the user. For instance, a user may point to a visible corner of a virtual cuboid object (such as a virtual air conditioning unit) seen in the augmented reality display. During this process, the corresponding real-world points determined along a ranging beam emitted from the handheld controller can be determined based on the measured distance as described above. The user may then choose to update the model position of the corner to the measured real-world point position. The BIM can then be updated such that the position of the virtual air conditioning unit matches the position of its corresponding real-world counterpart. Similarly, a user may approach the task from a real-world view. For example, they may use the handheld controller 1120 to point to a visible corner of an installed real-world air conditioning unit. The user may then choose to update the model such that the indicated virtual corner of the virtual air conditioning unit matches the measured corner in the real world.

[0180] Typically, the methods described herein can be used to align virtual objects with physical objects. In some cases, if the virtual object is a defined shape and / or constrained to a specific orientation, one or more points can be used to align the virtual object with the real world. For example, virtual points in the methods described above can include locations in virtual space that reference the definition of the virtual object, where a correspondence between real-world points and virtual points is used to position the virtual object relative to real-world points. For example, this method can be used to set a cube parallel to a wall on a construction site, or to place the cube at a corner of a wall. In some cases, the first and second points in the described examples are used to align a virtual object in virtual space with a physical location within a construction site.

[0181] In another example, a user can use a handheld controller to define a work area. The work area can be used to filter objects to be displayed in the augmented reality view (e.g., only portions of the BIM within the work area can be retrieved and rendered as part of the augmented reality view). In this example, the user can define the work area in a manner similar to indicator point 1322; for example, the user can select multiple corners of a polygon on the floor (e.g., four corners for a square or rectangular area), and then define a volume of a predetermined height as the work area. In its extension, the user can alternatively select points to define at least the height, width, and length of the work area. The work area can be used to set the rendering distance of a virtual space displayed within a head-mounted display (e.g., within the helmet 1110). This can help save power and / or simplify the augmented reality view by avoiding unnecessary rendering of virtual objects outside the work area.

[0182] In one example, the method described herein can be used to set the size of a virtual object. The user indicates a first real-world point within a construction site to be mapped to a first size reference point in the virtual space. Then, the user indicates a second real-world point to be mapped to a second size reference point in the virtual space. For example, these two real-world points could be opposite angles on a plane of an object or different points on an object. Since the positions of the real-world points can be measured using the method described above, the distance between these points can be accurately determined and used to set the size of the virtual object within the virtual space.

[0183] Typically, reference Figures 11 to 13 The examples described herein allow for the positioning of virtual objects in physical space using a handheld distance laser. Some examples allow users to accurately position and align virtual objects in 3D space using a handheld distance laser. The laser is used to better understand the 3D world and the distances to objects within it. Based on these measurements, virtual objects can be positioned and aligned in the 3D world. Measurement data from the ranging device can be passed to a digital processing system (such as a computational module formed by the integrated electronic subsystems described herein) for visualization of the digital data. This provides a more accurate and efficient method for processing virtual objects in the 3D world. Users can quickly and easily provide input to accurately define and / or interact with virtual 3D objects. The virtual object can then be "snap" or "clip" (i.e., aligned) into the actual physical space.

[0184] Additionally, refer to Figures 11 to 13 The methods described in the examples can be used to approve site inspections and reviews. For instance, in a manner similar to collaborative software coding tools, "issues" can be defined and associated with specific objects in the real world on the construction site. For example, if a beam is misaligned, a user can use a handheld controller to identify a real-world point on the beam and associate that coordinate with the issue log within the accompanying construction management software tool. The user can then annotate the real-world point or a corresponding virtual object, or both. Users operating the computing equipment on or off the site can then view the location associated with the issue and / or a given BIM element and schedule a fix.

[0185] In some cases, a reference system can be used when electronic ranging equipment is unavailable. Figures 11 to 13The example describes a method. For instance, instead of using an electronic rangefinder to measure distances to remote points, a point on a handheld controller or the user's body can be used to interact with the BIM model. For example, the upper single fork on the trident nose 830 can be used as a pointing tool. The user can use the upper single fork to touch objects within the surrounding environment and determine the tracking space location and the corresponding BIM location (which may in turn represent earthly or geocentric "real-world" coordinates). The conversion between the location determined in the tracking coordinate space (e.g., based on the tracking posture of the handheld controller in the tracking coordinate space) and the BIM or "real-world" location can be performed using a calibrated transformation as described. The location of a specific point on the controller (such as the upper single fork) can be determined in a manner similar to the modeled launch location, for example, using a known predefined transformation relative to the tracking origin of the handheld controller. The user can use the upper single fork to "touch" virtual objects seen on the display panel overlaid on the helmet and real-world objects seen through the display panel. The touch point can then be determined in one or more of the tracking coordinate space and the BIM coordinate space to allow interaction with virtual objects derived from the BIM data. For example, a user can touch a real-world point, and a set of virtual objects representing a feature to be installed at that point can be displayed in virtual space. Alternatively, a user can "touch" a visible object seen on the display panel of a safety helmet and then be told the object's real-world location (e.g., within a user interface used for augmented reality views). Users can also "draw" points within the BIM model space using a handheld controller; for example, a series of tracking points in the tracking coordinate space can be converted into a series of points in the geodetic BIM model space, which can then form "annotations" within the BIM model.

[0186] Exemplary BIM Data Processing

[0187] Figure 14 This is a flowchart 1400 illustrating an example method for preparing 3D Building Information Modeling (BIM) data for augmented reality applications. For example, this method can be applied to help display portions of the BIM within an augmented reality view on the display panel 222 of a safety helmet 210, such as... Figure 2DAs shown (and other examples described). In general, this method provides a way to filter or select 3D elements rendered as part of a BIM model displayed to the user in an augmented reality view. BIM models typically have hundreds (if not thousands) of 3D elements defined as part of a construction operation. Rendering all of these elements would be inefficient. Similarly, loading all the 3D elements of the BIM into the volatile and / or non-volatile memory of the integrated electronic subsystem (e.g., 260) of a safety helmet would require excessive memory and be time-consuming. However, manually selecting different 3D elements to include in an augmented reality view used for specific site inspections is also time-consuming. Therefore, it is desirable to provide a method for improving the management of 3D elements included in the BIM to help improve the efficiency of augmented reality systems providing augmented reality applications.

[0188] In construction, BIM is a digital representation of the physical and functional characteristics of a building. It serves as a shared knowledge resource for all operators involved throughout the building lifecycle, from design to construction and facilities management. Therefore, BIM can have thousands or even hundreds of thousands of different types and categories of model components or elements. Some example groupings of BIM elements include architectural components, structural components, mechanical components, electrical components, civil and site components, and interior furnishings and finishes. Brief descriptions of these components are provided below.

[0189] Architectural components shape the overall design and layout of a building. These components include walls, doors, windows, and partitions that define structural boundaries and spaces. Floors, ceilings, and roofs are elements that create horizontal planes and enclose the building, while stairs and ramps facilitate vertical circulation. Columns and beams provide structural support while also defining rooms and spaces. Elements that cover external features, such as facades, balconies, and landscaping elements, define the building's aesthetics and overall visual appeal.

[0190] Structural components ensure the stability and integrity of a building. In BIM, these elements include the foundation system, such as footings, piles, and ground beams, which transfer loads from the structure to the ground. Structural columns, beams, and joists are other elements that provide support and transfer loads between building components. Reinforced concrete and prestressed members enhance the strength and durability of the concrete structure. Trusses, braces, and flexural frames are elements that resist lateral forces, and floor slabs, roofs, and floor systems are elements that form horizontal surfaces that bear loads and provide usable space.

[0191] Mechanical components in BIM can form part of the operational equipment that provides a comfortable and functional indoor environment. Components include those related to HVAC systems, including air handling units, ductwork, vents, and diffusers. Components that form part of the piping system are also defined to ensure proper water distribution and waste removal. These components include pipes, fixtures, valves, and fittings. Components that form part of the fire protection system (such as sprinklers, pumps, and fire dampers) are defined as those used to protect the building and its occupants in emergency situations.

[0192] Electrical components in BIM are responsible for providing power, lighting, and communication functions within a building. These components include lighting fixtures, switches and sockets, as well as distribution boards, circuit breakers, and transformers. Wires, conduits, and cable trays are defined as those that carry current throughout the building. Components related to communication and data systems (such as network cabling, access points, and intercoms) are also defined to enable connectivity and information sharing. Components can also form part of a security system, including surveillance cameras and access control devices, which help ensure the safety and protection of building occupants.

[0193] Civil and site components in BIM address the building's surrounding environment and infrastructure, contributing to the building's overall functionality and sustainability. These components include site boundaries, topography, and contour lines, which define land and topographic features. Components may also include roads, sidewalks, and parking facilities. Components related to utility infrastructure, such as pipes and connections for water, sewers, and electricity, can be defined, and components related to drainage systems can also be provided.

[0194] Interior design and furniture components in BIM may be the last to be provided in a construction project. These components include furniture and equipment such as tables, chairs, and cabinets, as well as the definition of interior finishes, including paint, flooring, and ceiling materials.

[0195] These examples illustrate that BIM can include multiple 3D elements for display in augmented reality views. The data representing these 3D elements can reach many gigabytes. However, not all of this data needs to be displayed on augmented reality displays at different stages of a construction project.

[0196] Construction projects can have data defining activity-based construction plans. Activity-based construction planning is a method of organizing and scheduling construction projects by breaking them down into individual activities or tasks. This method focuses on identifying, prioritizing, and allocating resources to each activity to ensure timely completion and efficient resource utilization. Typically, the data defining activity-based construction plans is prepared manually by the project manager or management team. The data may include one or more of the following: a list of tasks to complete the project, such as site preparation, excavation, foundation work, rebar, concrete pouring, masonry, roofing, interior finishing, and landscaping; dependencies between tasks; the duration of each task (e.g., in days, weeks, or months); start and end dates; resource allocation, such as determining the labor, equipment, and materials required for each activity, including the quantity and type of resources, their availability, and cost; critical path data, representing the longest sequence of activities with the longest total duration, which determines the shortest project completion time; milestones during the project; and progress feedback, including fields updated based on reports as the project progresses. Data defining activity-based construction plans can be stored in spreadsheets, tag files, and / or databases. For example, activity-based construction plans can be prepared in software such as Primavera® from Oracle, Inc. (including the P6 Enterprise Portfolio Management application, which uses the ".xer" file format), Asta Powerproject from ProjectsAnalytics, Inc., or Microsoft Project from Microsoft, Inc. (using the ".mpp" file format). Activity-based construction plans can be provided in conjunction with BIM. In this case, 3D elements (e.g., assets and resources) can be manually assigned to different tasks as outlined in the plan.

[0197] Figure 14 This method provides a way to automatically associate 3D elements of BIM with different tasks defined in the planning data of an activity-based construction schedule. In this method, historical plan data is processed to learn the association between the planning data and the element data that defines the 3D elements of the BIM. This method uses a machine learning system. Therefore, new planning data can be fed to the machine learning system to identify candidate 3D elements to be assigned to unseen tasks in the planning data. This provides a fast way to assign 3D elements to different tasks. Users of the augmented reality application can then view a task-based augmented reality view to filter the BIM elements that need to be rendered. Tasks can be selected for the augmented reality view manually (e.g., via an augmented reality user interface or before the user puts on a hard hat) and / or automatically (e.g., based on the current date and time and the start and end times of the task definition).

[0198] Figure 14 The method begins with step 1412, which involves obtaining planning data defining an activity-based construction schedule. The activity-based construction schedule includes multiple tasks to be performed as part of the (current) construction project. The planning data may include one or more of spreadsheet data, markup language data, and database data. This step may include loading all or a portion of the planning data into memory. In step 1414, the method includes obtaining element data representing a set of 3D elements defined within BIM data. This step may include loading the definition of that 3D element set into memory. Each construction project may have its own BIM (although multiple construction projects may also use a common BIM). Step 1414 may load only a portion of the complete definition into memory at a time, for example, it may include loading the name and pointer of each available element in the set, where the pointer indicates data of the element in local and / or remote non-volatile storage. The precise definition of the element data may depend on the BIM specifications and the software used to create, define, and / or load the 3D elements. Therefore, it can vary between different implementations; however, import routines can be defined based on the BIM specifications in any particular application and / or a standardized BIM configuration that is frequently used.

[0199] Figure 15A An exemplary user interface 1500 is shown, in which a user is viewing exemplary 3D element data 1502 of a wall 1504. In this example, the 3D element data includes: element type; element identifier; data defining the geometry of the element, including coordinates of the extent, height, and thickness; data defining the materials used; data defining the properties of the materials used; data defining structural properties; data defining the layered structure of the element; data defining the external and / or internal finishes of the element; and relevant data indicating the manufacturer (and in some cases defining the product type and / or specifications). For augmented reality views, the element is rendered using at least the data defining its geometry. In some cases, for a specific project and / or project task, the element may be placed at a specific location within a construction site. This placement information may also be present within the 3D element data. Figure 15A A view of a set of basic data fields is shown, which can be represented in a machine-readable manner (e.g., as database fields, spreadsheet cells, and / or tag entries).

[0200] One or both of the planning data used for activity-based construction planning and the element data representing a set of 3D elements may include specific, customized information relevant to the construction project in question. However, one or both of the planning data and element data may reuse existing portions of the data. For example, the set of 3D elements may be used for multiple construction projects (e.g., HVAC units, piping, etc.) but may also be specifically tailored to each project (e.g., walls in a particular project may have specific location and geometry parameters for each construction project). Typically, the planning data and the set of 3D elements have aspects unique to each construction project. In this approach, the planning data may relate to multiple specific tasks to be performed as part of an upcoming or ongoing construction project, and the set of 3D elements may include one or more of a set of 3D elements already defined for the construction project and / or pre-existing 3D elements that can be assigned to the construction project.

[0201] Return to Figure 14 In step 1416, at least one task associated with the construction project defined in the planning data is selected as a "given" task. In one case, the method can be performed before or during the given task. In another case, the method can be performed for all tasks before the construction project begins. In step 1416, a subset of the set of 3D elements in the element data obtained in step 1414 is assigned as a candidate element set to the given task. Candidate elements may include elements that are later confirmed through one or both of user selection and further processing. In one case, candidate elements can be automatically selected for assignment. In other cases, the user can confirm each candidate element or set of candidate elements. The assignment process uses a machine learning system to process the portion of the planning data associated with the given task and the element data, and to match elements within the element data with the given task. Different methods for performing assignment are discussed in more detail below. Typically, assignment uses assignment data configured based on a training set of the planning data, where 3D elements are assigned to tasks within the planning data. For example, assigned data may include one or more of the following: rule sets; neural network weights (e.g., parameters); decision tree weights; and frequencies and / or probabilities (e.g., for Bayesian methods).

[0202] In step 1418, candidate elements are provided for generating a task-specific augmented reality view of the construction site associated with the construction project, where the task-specific augmented reality view is associated with the given task. For example, candidate elements can be reviewed and confirmed by the user (e.g., using a computing device and display) before a site visit and / or during a site visit when the user is wearing the hard hat of the previous example. In some cases, candidate elements (or confirmed candidate elements) can be loaded only from the set of all element data used to provide the augmented reality view of the construction site. This then means less data needs to be synchronized with the integrated electronic subsystem of the previous example to provide the augmented reality view.

[0203] Figure 15B It shows that it can be displayed as execution Figure 14 Example user interface 1508 is part of the method. In one case, user interface 1508 is displayed on a desktop or laptop computer before a site visit is conducted using one or more of the augmented reality components as described herein. In another case, user interface 1508 may form part of an augmented reality user interface displayed on a display panel of a user wearing the safety helmet of the previous example. Figure 15B In the middle, the user interface 1508 displays the "Explorer" screen 1510, in which the user can view at least a portion of the current construction project's activity-based construction plan 1520 and 3D elements that have been assigned to each of multiple tasks. Figure 15B In this example, an activity-based construction plan 1520 is shown as a table, with rows indicating individual tasks within the plan. The activity-based construction plan 1520 may show all or a subset of tasks for a construction project and can be scrolled and / or filtered using known methods. In this example, each example task includes: an identifier 1522; a task description 1524; a task start date 1526 (which may also include time in some examples); a task end date 1528 (which may also include time in some examples); and at least one link to the assigned set of 3D elements for task 1530. The assigned 3D elements may include those using... Figure 14 The method assigns one or more of the following: unapproved candidate elements, approved candidate elements confirmed by the user, and elements manually assigned by the user. Figure 15BThe example illustrates three sample tasks: "Foundation" (i.e., constructing a set of foundations); "Steel Beams" (i.e., placing and securing a set of steel beams for the structure of a building); and "Concrete" (i.e., pouring concrete to form the structure within the building). It can be seen that a task can have many associated 3D BIM elements. In one case, the "Elements" column 1530 may initially be unfilled, and the user can select a row and activate a user interface function (e.g., via click, touch, button press, or key press) to perform the task. Figure 14 The method assigns candidate elements to the tasks identified in the row.

[0204] Figure 15B A preview of at least a subset of the elements assigned to each task is also provided. This preview can change as the user clicks on different rows of the activity-based construction schedule to display elements associated with different tasks. In one case, the user can select multiple tasks at once and display elements associated with a set of selected tasks. Figure 15 shows a concrete slab 1540, as well as pipes 1542 and steel support columns 1544.

[0205] exist Figure 14 In some implementations of the method, providing candidate elements includes: displaying a list of candidate elements associated with a given task to a user; receiving from the user a confirmed selection of candidate elements to be used in a task-specific augmented reality view for the given task; and assigning the confirmed selection of candidate elements to the given task. Each candidate element can be confirmed and / or rejected individually, and / or multiple sets of candidate elements can be confirmed and / or rejected jointly. The data defining the confirmed selection of candidate elements and the given task can be used to configure assignment data for other tasks; for example, the confirmation and / or rejection of the data can itself be used as part of the training data for training a machine learning system.

[0206] For later use, the method may include viewing an augmented reality view of a construction site via a head-mounted display; selecting a given task from a plurality of tasks using an augmented reality user interface displayed within the head-mounted display; and populating the augmented reality view of the construction site with confirmed candidate elements within a virtual layer of the augmented reality view. The head-mounted display may include a display panel 222 such as a hard hat 210 (and as shown in other figures).

[0207] It can be based on the element name, element type, and one or more element attributes associated with the assigned 3D element (e.g., Figure 15AThe assignment data is configured using one or more of the exemplary data fields shown. In one case, the element data for each 3D element is converted into a numeric vector form (e.g., a vector of single-precision or double-precision floating-point values ​​of length 256-2048), and the planning data for each task is also converted into a numeric vector form (e.g., with the same or different lengths). A similarity metric can then be used to compare a given task and a given 3D element. For example, the cosine similarity of any two given vectors (e.g., a latent vector pair of task data for a given task and element data for a given 3D element) can be computed to provide a normalized similarity metric between 0 and 1. A threshold can then be applied to select candidate elements as those with a similarity metric higher than the threshold. The conversion to numeric vector form can be performed based on a trained neural network; for example, two feedforward neural networks can be provided to compute the corresponding task embeddings and element embeddings. A neural network computing the embeddings can be trained to maximize a similarity metric for 3D elements assigned (and / or approved) for a given task. For example, training data may include task data, element data, and scores of 0 and 1 or -1, 0 and 1 representing “ideal” similarity. The neural network can then be trained based on a loss calculated as the difference between the similarity computed during training and the “ideal” similarity. In one case, text values ​​can be converted to initial numerical values ​​based on a dictionary lookup (e.g., the element types of all element data can be parsed to generate a type lookup dictionary, where each distinct type in the dictionary is assigned a number based on a hash and / or index in the dictionary). Numerical values ​​in the element data or plan data can be performed, normalized, and / or embedded based on values ​​and / or ranges. In some cases, textual descriptions of one or both of the task data and element data can be tokenized and converted to lexical values ​​(e.g., using known tokenization methods). In one scenario, pre-trained embeddings (such as FastText- or BERT-based embeddings) can be used to convert words in a data field into vector equivalents. In another scenario, associations can be based on term frequency, for example using methods such as Term Frequency-Inverse Document Frequency (TD-IDF), where the importance of terms used in one or both of the task data and element data can be weighted based on term frequency, such as by down-weighting based on their frequency of use on the entire dataset. The TD-IDF metric can be computed for text data found in elements of different "corpora" of task keywords.

[0208] In one scenario, element data is converted into numerical vector form, and then clustering is applied in a multidimensional space to determine if task-based clustering can be identified. In another scenario, 3D elements can be associated with specific areas, rooms, and / or sites defined explicitly or implicitly in the planning data. For example, a task describing “assembling a data center” can be processed to extract the noun phrase (“renovating a data center”), and 3D elements assigned to that task in historical data can be counted. When a new task with the same noun phrase (“data center”) is received, frequencies can be processed, and frequencies above a certain normalization threshold can be included as candidate elements. In some cases, BIM data of 3D elements can be processed to determine hypernyms of part names, types, and other string labels. For example, a lookup service such as WordNet can be used, and / or a custom dictionary can be defined. In this case, a probability distribution representing the joint probability of task items and element items at different semantic levels can be constructed. In addition to or as a supplement to terminology, n-gram probabilities can be calculated based on descriptive string fields and used to determine relevance. Typically, semantic and BIM information can be used to learn the aforementioned associations.

[0209] The training process can include data preparation, model training, and inference phases. Past planning data and their assigned 3D elements can be combined into a single dataset. Feature extraction can be performed. For example, features based on one or more of the following attributes—task type, duration, material, geometry, and structural properties—can be extracted and represented in a consistent and comparable format. During model training, a supervised learning algorithm can be selected (e.g., from a group of decision trees, random forests, support vector machines, or neural networks) and trained using the training dataset. A suitable model can be selected based on initial training results and evaluation. Model configuration and hyperparameters can be selected during the iteration process based on the training results. The trained model can then be applied to new planning data to predict suitable candidate 3D elements. For example, when training a decision tree, task-related features (such as task type, duration, and required material) and 3D element features (such as geometry, material properties, and structural properties) can be used as input attributes to the decision tree, and the output can include binary values ​​or probability values ​​indicating assignment or assignment probability. This can then be used to predict the assignment probability of unseen task and element pairs.

[0210] In one scenario, assignment data can be configured based on the fact that certain types of industries and / or companies exhibit recurring similarities in the layout of buildings. Therefore, there are often definable or learnable patterns that enable prediction of 3D elements. For example, industry types (“technical infrastructure,” “water facilities,” or “residential”) can be extracted from and / or inferred from planning data as classifications performed on the planning data (e.g., using a trained machine learning model). An initial candidate set of 3D elements can then be retrieved using the association between industry types and the commonly used set of 3D elements (e.g., in the form of probabilities or normalized frequencies or learned associations). Similarly, construction tasks within planning data can refer to locations (e.g., room or site locations). These locations can be extracted, and the association between these locations and the set of 3D elements can be determined. Furthermore, or alternatively, a certain type of object may exist within the planning data. This can be extracted and used to map to specific 3D elements (e.g., a task could have a description “install sprinklers in room E” - then “sprinklers” can be extracted as a parsed object type and mapped to 3D elements representing sprinklers to be returned as a candidate set).

[0211] In one scenario, images and / or videos from construction sites and / or completed buildings can be used to determine associated data. In another scenario, images taken during or after a specific task can be processed to associate 3D elements with planned task data. For example, objects within an image can be identified and mapped to specific 3D elements to form associations. Images and / or videos can be segmented and / or classified using neural network architectures. Images and / or videos can be used using methods such as... Figure 2A The camera assembly 230 shown is used to capture images, and / or the images can be obtained from a database of historical site inspections where no 3D element data is available.

[0212] This method may include the following general steps: receiving input data from the user in the form of 3D elements and planning data; learning associations between 3D elements and activities (i.e., tasks) based on semantic and BIM information, such as element names, types, and other attributes provided by the user; suggesting new pairings (e.g., providing candidate elements) based on the learned associations; and reviewing and selectively confirming (or rejecting) the suggested pairings of activities / tasks and elements. The described examples offer several advantages compared to traditional methods for linking 3D elements to activity-based construction plans. For example, they automate the process, reducing the time and effort required to complete the task. Furthermore, they leverage previous construction data, improving the accuracy and reliability of associations between 3D elements and activities. Finally, they provide a user-friendly interface to view and confirm suggested associations. The described examples address the problem that linking 3D elements to activity-based construction plans can be a challenging and time-consuming process, especially when a large number of elements are involved.

[0213] In some of the examples described in this article (including) Figures 11 to 17 In those examples, the functions and / or methods may be implemented by a processor (such as a processor forming part of the integrated electronic subsystem or another electronic device described herein), the processor being configured to load instructions stored in a storage device into memory for execution. In use, the functions and / or methods described herein are implemented by one or more processors executing instructions (such as machine code and / or compiled computer program code). Although this example is presented based on some local processing, it will be understood that in other implementations, the functions may be distributed across a set of local and remote devices, for example, via a network interface. The computer program code may be prepared in one or more known languages, including custom machine or microprocessor code, C, C++, and Python.

[0214] Exemplary model alignment method

[0215] Figures 16A to 16L Some stages of an exemplary model alignment method are illustrated. This method aims to provide a simple, computationally efficient way to align at least a portion of a building information model with a view of a construction site to provide an augmented reality view (e.g., overlaying spatially aligned aspects of the building information model onto a view of the construction site). Figures 16A to 16K The example phase of the method is illustrated schematically, and... Figure 17 The corresponding flowchart of this method is shown in the figure.

[0216] This exemplary method and reference Figure 12B and Figure 13The described example methods have overlap. This exemplary method can use a combination of the safety helmet 1110 and the handheld controller 1120 as previously described. In other cases, this exemplary method can alternatively be implemented using a handheld mobile device with an integrated screen, such as a tablet or smartphone.

[0217] Typically, this example method measures points that form part of a set of surfaces in the real world (i.e., on a physical construction site) and then uses those measurements to determine the transformations to be applied to the Building Information Model (BIM) to align that model with a local coordinate system used to track the pose of a display showing an augmented reality view. This method can be used to avoid the need for a set of locations (e.g., a set of control points as described above) that have known positions in both the BIM and the real world. This then reduces the need for measurement tools to measure the measurement markers that form the control points, the need to define those control points in the BIM, and / or the need for the user to actively measure those same measurement markers using a tracking system. Furthermore, using surfaces instead of points introduces an averaging element that results in more robust alignment. This method also offers greater robustness compared to computationally more computationally intensive and sensitive image processing algorithms.

[0218] Figure 16A A user is shown with an augmented reality helmet 1610 and a handheld controller 1620. Helmet 1610 may include the helmet of the previous example (e.g., as shown in Reference 1620). Figures 2A to 7C (As described), or may include another device. The handheld controller 1620 can be used to interact with a virtual representation of a construction site viewed by a user with a head-mounted display (e.g., a user wearing a hard hat 1610). In this case, the handheld controller 1620 is detached from the hard hat 1610. Figures 16D to 16F and Figures 16H to 16J In the example, the handheld controller 1620 includes a set of sensors for a position tracking system (e.g., Figures 8A to 8D The photoelectric sensor 812) and electronic rangefinder (e.g., Figure 8D (A laser measurement device having a transmitter 842 and a receiver 844). Therefore, the handheld controller 1620 in this example is similar to the reference. Figures 11 to 13 The handheld controller 1620 is described in the example. However, in some cases, the electronic rangefinder can be omitted from the handheld controller 1620. (See reference...) Figure 16LAn example illustrating one of these scenarios is provided. While this example is described with reference to a helmet and a separate handheld controller, the general approach can also be applied using only augmented or mixed reality head-mounted devices and / or mobile devices (such as phones or tablets) with cameras and screens. In one case, the method can be applied using a handheld device including a smartphone with a built-in LiDAR sensor and / or infrared depth sensor. In these latter cases, augmented reality images can be displayed on the screen of the handheld device, for example, as an overlay of video feeds from one or more cameras and / or as a composite video based on said video feeds.

[0219] exist Figure 16A In this scenario, the user uses the augmented reality display inside the safety helmet 1610 to view the augmented reality representation of the building information model 1630. The user also uses a handheld controller 1620 to interact with the augmented reality environment. Figure 16A and Figure 16B In this system, a user can use a handheld controller 1620 to select a specific building information model 1630 and then filter the view of that model. For example, a user can use an augmented reality interface (viewable with an augmented reality display) to select a specific building information model from a list of available building information models. In other cases, the building information model can be determined automatically, for example, it can be pre-loaded for specific site inspections and / or selected based on GPS location. Figure 16A and Figure 16B As shown, the building information model 1630 is displayed at a scaled-down ratio within a cuboid containing volume 1632. The size of this containing volume 1632 can be adjusted, as indicated by arrow 1634, to view specific portions of the building information model. Although this example shows a cuboid containing volumes for scaling and filtering the building information model, other implementations may only show the model and / or use different interface methods to view and / or filter portions of the model. The volume also does not need to be a cuboid, but can include any polygonal containing volume.

[0220] exist Figure 16BIn the example, the building information model is filtered to show three structures within the model. These include the rear wall 1640, the right side wall 1642, and the left side wall 1644. Each of the three structures has multiple flat surfaces. It should be noted that this example has been simplified for illustrative purposes; an actual building information model could include many more structures—for example, a door or window opening could include multiple surfaces forming a recess for a door or window. The surfaces also do not need to be planar (e.g., they could include cylindrical columns with measurable geometry). The user can select one of the faces of the accommodating volume 1632 and move it within the augmented reality view to focus on the filtered set of structures within the building information model 1630. For example, only a portion of the building information model present in the accommodating volume 1632 may be shown and / or the building information model may be cropped to fit within the accommodating volume 1632.

[0221] Figures 16C to 16F The process of selecting a surface 1650 using a building information model 1630 and then measuring the corresponding points on the real-world surface at the construction site is shown. Figures 16G to 16J A similar process is shown for another surface 1680. Using the correspondence between measurement points in the building information model and the selected (virtual) surface, the building information model can be aligned with the tracking posture of the safety helmet 1610, thus aligning the augmented reality view of the building information model with the user's view of the construction site. This aligned view is... Figure 16K As shown in the image.

[0222] Return to Figure 16C The user views an misaligned augmented reality view of the building information model. For example, structures 1640 to 1644 may have a reduced scale and be rotatable compared to any actual structure on an external construction site. In the misaligned augmented reality view, the user selects surface 1650 of one of these structures. This can be achieved using a handheld controller 1620 to navigate the user interface in virtual space, as indicated by arrow 1652. In this case, the front of structure 1640 is selected. Surface 1654 of the other structures is not selected.

[0223] After making a selection, the user then navigates to a view of the external environment. This is in Figures 16D to 16F As shown in the diagram. The external environment can be viewed by hiding the containment volume 1632, that is, by hiding the virtual aspect of the augmented reality view of the building information model, allowing the user to view the outside world through the transparent display panel 222. In the case of using a mobile device, the camera feed can be displayed on the mobile device's screen. The external environment has multiple structures corresponding to the structure of the building information model. These include a rear wall 1660 and two side walls 1662 and 1664. After selecting the virtual surface 1650, the user then... Figures 16D to 16FThe position of the corresponding real-world surface 1661 (i.e., the front plane of the rear wall 1660) is then measured. Figure 11 and Figure 12A The method can be used to measure the real-world position of point 1666 on real-world surface 1661 (e.g., using the tracking attitude and distance measurement 1668 of the handheld controller 1620). Alternatively, if the handheld controller does not include an electronic rangefinder, point 1666 can be measured by positioning the tip of the handheld controller 1620 on the real-world surface (e.g., engaging the upper single fork of the trident nose 830 with the rear wall). Figure 16L An example of this direct measurement is shown, in which the user aligns the tip of the handheld controller 1620 with point 1666. Since the handheld controller 1620 is tracked by a tracking system (which is also used by the helmet 1610), it can be located within a tracking coordinate system. In other embodiments, the handheld controller may be tracked by a separate tracking system of the reference helmet (e.g., an infrared tracking system mounted in the helmet). It should be noted that point 1666 does not need to be any particular or specific point on the real-world surface 1661; it can simply be a randomly selected first location forming part of the surface. When using a mobile device instead of a handheld controller, point 1666 can be measured using a LiDAR camera on the mobile device and / or by moving the mobile device to the point and holding the mobile device in a predetermined orientation.

[0224] exist Figure 16E and Figure 16F In the diagram, the user indicates and measures two additional points, 1670 and 1672, on surface 1661. This forms a triangle. These measured points can be shown with virtual annotations within the augmented reality view. For example, they can be highlighted with yellow circles, and virtual lines can be drawn to connect these points, as shown by the dashed lines in the diagram. Figures 16C to 16F Following this process, the set of three measurement points in the tracking coordinate system can be assigned to the model surface 1650. As described with reference to the previous example, the tracking coordinate system is used by a tracking system that tracks the position and / or orientation of the helmet 1610 and the handheld controller 1620.

[0225] exist Figures 16G to 16J In the middle, repeating for another model surface Figures 16E to 16F The process. In Figure 16G In the virtual scenario, the user uses a handheld controller 1620 to select a surface of the form 1680 (virtually shown) on the inner surface of the right wall structure 1642. Other surfaces 1684 of other structures are deselected. In the real world, the user moves within the construction site to align themselves with the corresponding real-world structure 1662. Figure 16HIn the middle, the user turns right as indicated by arrow 1686. Then, the user follows... Figures 16H to 16J In Figures 16D to 16F A similar process was used to measure the positions of three points 1690, 1692, and 1694 on the real-world surface 1691.

[0226] repeat Figures 16E to 16F and Figures 16G to 16J The process illustrated continues until a suitable number of real-world surfaces have been measured to allow for explicit alignment between the model coordinate system and the tracking coordinate system used for the building information model. In the simplest case, a building information model with a single wall structure may only need to measure three points on a single wall surface to align the building information model. However, more complex building information models, typically with multiple structures and surfaces, may require measuring multiple real-world surfaces to allow for explicit alignment. In one case, three vertical surfaces (e.g., two surfaces 1661 and 1691 and the floor or ceiling) can provide robust alignment. Increasing the number of surfaces can remove matching and scaling ambiguities depending on the building information model. The number of surfaces to be measured can be dynamically configured for each alignment routine based on the building information model and / or existing measured surfaces. For example, if there is not enough data to calculate the alignment, the user can be prompted to measure another surface via an augmented reality user interface.

[0227] As described previously regarding the reference control point set, the coordinates of three or four points in each coordinate system are sufficient to derive the transformation mapping between the model coordinate system and the tracking coordinate system. This transformation allows the rendering of the building information model to be aligned with the position and orientation of the display providing the augmented reality view. Four points provide a more robust mapping and allow for proper scaling. The parameters of the transformation matrix (e.g., rotation and translation parameters) can be derived by solving a set of equations defining selected surfaces in the building information model and measured surfaces in the real world. In some cases, these parameters can be derived using known optimization techniques (e.g., least squares). In one case, the normal vectors of each set of surfaces (BIM and measured) can be defined as two matrices: a matrix N with the BIM surface normal vectors arranged in the columns of that matrix, and another matrix N' with the measured surface normal vectors arranged in the columns of that matrix. Matrix products (e.g., N'') can be computed. T The singular value decomposition (U, ∑, V) of N' (where N' represents the covariance of the two sets of normal vectors) is used to derive the rotation matrix (e.g., R = VU). TThe translation vector can be derived by comparing two corresponding points in each coordinate system (e.g., by subtracting the origin of another coordinate system rotated by R from the origin of one coordinate system). The transformation matrix can then be determined from the rotation matrix and the translation vector. Methods for calculating the transformation matrix can be based on solutions to the Kabsch algorithm, the Wahba problem, or the orthogonal Procrustes problem.

[0228] Figure 16K A virtual view of an aligned Building Information Model (BIM) is shown. In this case, virtual structures 1640, 1642, and 1644 are aligned with their real-world counterparts 1660, 1662, and 1664, respectively. Once the transformation matrix is ​​determined to map between the BIM coordinate system and the tracking coordinate system, a suitable 2D projection of the BIM aligned to the plane of the augmented reality display can be calculated. The transformation matrix can then be used to render the aligned augmented reality view of the BIM as the user explores the construction site. This aligned augmented reality view can be used for site inspections and guiding construction work, among other things.

[0229] Figure 17 Method 1700 is shown for aligning a building information model (or a portion thereof) with an augmented reality view of a construction site. Method 1700 corresponds to... Figures 16A to 16K The actions shown can be performed by a processor within the helmet (e.g., forming part of the integrated electronic subsystem 660) or by a processor within another mobile device.

[0230] At box 1712, the building information model is obtained. In one scenario, the user can use an augmented reality interface to select and load the building information model stored in the storage device of the safety helmet 1610. In another scenario, a single building information model can be pre-stored for a specific site operation and can be loaded into memory (at least partially) when the safety helmet 1610 is activated. In yet another scenario, a suitable building information model can be automatically downloaded from a remote server device based on the tracking position of the mobile device displaying the augmented reality view. At this stage, the building information model is not aligned with the augmented reality view. For example, the building information model can be defined within a model coordinate system with a defined origin; in the augmented reality view, the origin of the model coordinate system can be mapped to the tracking coordinate system using default transformation matrices (e.g., based on a unit rotation matrix and view-based translation).

[0231] At box 1714, the building information model can be filtered to allow users to better see different objects and / or structures within the model. For example, the following can be performed: Figure 16A and Figure 16BThe sequence is shown. Filtering can involve cropping and / or hiding portions of the building information model based on the movable containment volume visible in the augmented reality view. In other cases, certain objects and / or structures can be selected from a list displayed in the augmented reality view.

[0232] At box 1716, selection of a model surface in the building information model is received. For example, a user can use a handheld controller 1620 to select the surface of a specific object or structure in the building information model. The model surface can be a plane of a specific object and / or structure. Figure 16C and Figure 16G An example of this step is shown. The model surface can also be either the ceiling or the floor. The model surface can be determined by identifying the intersection of light rays projected from the tip of the virtual twin of the handheld controller. When using a mobile device, the user can use the touchscreen to select the model surface to be displayed as part of the augmented reality view.

[0233] At box 1718, a measurement of a point in the external real world is received. For example, according to Figure 11 and Figure 12A The user can use the handheld controller 1620 to indicate a remote point on a real-world surface. An electronic ranging device can then determine the distance from the handheld controller to that remote point. This distance (given the launch position) and the tracking attitude of the handheld controller can then be used to locate the remote point within the tracking coordinate system. In other cases, such as... Figure 16L In the cases shown, the handheld controller 1620 can be physically moved such that a known point on the controller indicates a point that is then measured using the tracking attitude of the handheld controller and the known design distance of the handheld controller.

[0234] like Figure 17 As shown, box 1718 can be repeated m times to measure the positions of m points on the real-world surface corresponding to the model surface selected in box 1716. Therefore, the positions of these points in the three dimensions of the tracking coordinate system are known. In one case, m is greater than or equal to 3, such that by repeating box 1718, the user indicates a triangular region on the real-world surface. This in Figures 16D to 16F and Figures 16H to 16J As shown in the diagram. At box 1720, a check is performed to determine whether the indication of a single surface in the construction site environment has been completed. For example, this could include checking whether the required number of points have been measured. It could also include certain verification routines. For example, if the selected model surface is horizontal or vertical within the building information model, and the plane formed by m points deviates from horizontal or vertical by a predetermined threshold, the user can be prompted to repeat the measurement.

[0235] like Figure 17 As shown, boxes 1716, 1718, and 1720 are repeated n times to provide measurement data for a set of n surfaces. In one case, n is greater than or equal to 3. The selection of model surfaces can be constrained at box 1716, allowing for the selection of different sets of model surfaces. For example, a set of vertical surfaces and / or a set of surfaces spaced more than a predetermined distance threshold can be used to achieve more reliable alignment. In other cases, the user is free to select surfaces and repeats boxes 1716, 1718, and 1720 until sufficient data is collected to provide robust alignment. A check on the number and form of surfaces can be applied at box 1722 until the desired number of model surfaces has been selected and the corresponding real-world surfaces have been measured (i.e., such that there exists a set of model surfaces defined in the model coordinate system and a set of real-world surfaces defined in the tracking coordinate system).

[0236] At box 1724, data defining corresponding measurement points on the selected model surface and real-world surface are processed to match the indicated surface in the construction site environment with the model surface. This could include, for example, solving a set of equations defining individual planes or normal vectors of these planes, as described above. Matching at box 1724 could include calculating a transformation matrix that includes rotation and translation vectors. The scale factor can be determined by comparing the distances between paired surfaces in the model and paired measurement surfaces. In this way, at box 1726, the building information model can be mapped to a tracking coordinate system for displaying an aligned augmented reality view. Depending on the matching at box 1724, a transformation matrix can be defined in either direction (e.g., by swapping data points in the equations to be solved or by calculating the inverse transformation matrix). The mapping between the tracking coordinate system and the model coordinate system can allow for display within the building information model (e.g., where users and / or handheld controllers can be shown). Figure 16A and Figure 16B (In the augmented reality view shown) the object being tracked within the tracking coordinate system.

[0237] Other examples and information

[0238] To display augmented reality information, a position tracking system such as position tracking system 100 can be used. It should be noted that in other examples, other position tracking systems such as optical tracking systems can be used, and equivalents in those systems (such as active and / or passive optical markers) can replace photoelectric sensors. Combinations of position tracking systems can also be used, for example, as described in WO 2022 / 167505 A1, which is incorporated herein by reference. For example, data from a camera component can be used in methods for fusing multiple positioning systems, as described in the aforementioned publications. Figure 1A and Figure 1BIn the examples, multiple sensor devices on the exemplary safety helmet and the exemplary handheld controller, for example, use a set of tracking beacons as described herein to track the position of the safety helmet and the handheld controller within a tracking volume defined by a location tracking system set up at a construction site. While the examples include specific sensor devices for a particular positioning system, these are provided for illustrative purposes only; implementations can utilize any type or technology for the positioning system, including known or future “off-the-shelf” positioning systems. One or both of the safety helmet and the handheld controller may also include one or more inertial measurement units (IMUs) of the type found in virtual reality and augmented reality head-mounted devices, comprising a combination of one or more accelerometers and one or more gyroscopes. The IMU may include an accelerometer and a gyroscope for each of pitch, roll, and yaw modes. These can be used to supplement short-term location tracking in conjunction with longer-term positioning provided by the location tracking system (e.g., it is well known that IMU drift caused by the limited accuracy of IMU sensors renders any position tracked using the IMU unavailable after a few seconds). In some variations, eye-tracking devices may also be used. These may not be used in all implementations, but can improve the display in some cases at the cost of additional complexity. The example described in this article was implemented without an eye-tracking device.

[0239] In some examples, such as in Figure 2A One or more camera devices in the illustrated camera assembly 230 are arranged to provide positioning data. For example, one or more camera devices may include one or more camera devices with a wide field of view (e.g., in the horizontal range) to capture images of the area surrounding the helmet 210. Here, the term "wide" may refer to a field of view greater than 90 degrees in the horizontal direction. The quality of the camera devices can be selected based on tracking accuracy. For example, a relatively low-resolution camera device may be able to capture images that allow determination of the relative position and orientation of objects within the line of sight of the helmet 210 and / or the helmet. In some cases, it may provide, for example... Figure 2AMultiple camera devices are shown. Images from one or more camera devices can be provided to one or more computer programs, for example, running within an integrated electronic subsystem and / or within a set of distributed computing devices, for detecting objects within the field of view and determining one or both of the position and orientation of the object and / or the helmet 210. For example, firmware or other computer program code can be loaded into memory and executed by the processor of the computing module to determine the pose of an object characterized within the images captured by one or more camera devices. In some cases, one or more camera devices may include video devices arranged to provide a stream of images (e.g., video frames). Object detection and determination of one or both of the object's position and orientation can be performed on one or more frames provided from the stream. Processing can be performed on each frame or every n frames (e.g., depending on computing resources). In one case, the conditional processing pipeline may include detection and pose determination stages, which may be sequential. The detection stage may include a speed-optimized function that may run per frame or every m frames, where m is chosen to provide a relatively large number of frames per second (e.g., 5-20) of processing. In response to the detection of an object within a frame, the frame can be passed to pose determination to determine the object's position and orientation within the frame. Therefore, the detection level can act as a filter, allowing pose determination to be performed conditioned on the detection of an object in the vicinity.

[0240] To perform object detection and / or obtain SLAM localization data for the helmet, the pose determination level of the computation module can use one or more frames to determine the position and orientation of objects visible to camera component 230. For example, the pose determination level may include computer vision functions provided by an image processing library, configured to determine the pose of the localized object. This pose determination level may differ from any pose determination performed on the helmet 210 using tracking beacon 102 based on data from photoelectric sensor 212. The pose determination level may receive localization data indicating the position of the localized object within an image (e.g., in the form of a bounding box or centroid). The pose determination level may determine the pose relative to one or more camera devices on the helmet 210. For example, given the positions of n 3D points on the object and their corresponding points within the captured image, the pose determination level can solve the perspective n-point (PNP) problem. One or more intrinsic parameters of the camera devices (e.g., focal length, optical center, and radial distortion parameters) may be provided to the pose determination level (e.g., in the form of data loaded into memory). Alternatively, these may be approximated during image acquisition. Any inherent parameters of one or more camera devices used can be measured and stored as part of the setup or calibration phase before use or factory calibration loading. The pose determination level can use one or more of the solvePnP or solvePnPRansac functions provided by the OpenCV library. Alternatively, the pose determination level can utilize a trained deep neural network. In one case, both the detection level and the pose determination level can be combined into an inference process for a deep neural network that receives image data frames (e.g., grayscale, YUV, or RGB) and outputs one or more 6-DOF poses (i.e., 6-parameter variables) for the detected object. Such a deep neural network can be based on a convolutional neural network preceding a feedforward neural network.

[0241] Exemplary terms

[0242] This section introduces the various clauses. Depending on the specific patent application, these may or may not be protected.

[0243] In a first aspect, a multi-layered safety helmet is provided. This can be provided in a safety helmet having at least one integrated electronic subsystem. The multi-layered safety helmet improves user safety and comfort. In a second aspect, a safety helmet with impact-resistant foam is provided. The impact-resistant foam can be positioned between two layers of the multi-layered safety helmet. The impact-resistant foam helps improve safety, especially against side impacts. In a third aspect, a deformable ventilated connector for a safety helmet is described. Again, this can be used with a multi-layered safety helmet having at least one integrated electronic subsystem. This third aspect can improve airflow within the safety helmet and improve user comfort during prolonged use (e.g., site visits). In a fourth aspect, a safety helmet with an integrated electronic subsystem includes multiple battery connection interfaces for connecting multiple removable batteries, wherein power can be provided by one of the multiple batteries while another of the multiple batteries is replaced. This enables continuous use of the integrated electronic subsystem, for example, in the form of an augmented reality system, without recalibration. In a fifth aspect, a removable battery housing for removable batteries in a safety helmet with an integrated electronic subsystem is provided. The removable battery housing can include a unique securing mechanism for improving the ease of battery replacement. In a sixth aspect, the positioning of multiple battery connection interfaces on a helmet with an integrated electronic subsystem is configured to lower the helmet's center of gravity, improving stability and comfort on the user's head. In a seventh aspect, a component kit for augmented reality applications on construction sites is provided. This kit may include various combinations of the different aspects discussed herein, as well as one or more of the following: a universal rechargeable battery, a handheld controller, a tracking beacon, and a battery charging station. In an eighth aspect, a bracket height adjustment mechanism for a helmet is provided. The eighth aspect may also include an adjustment method. The eighth aspect can improve the configurability of helmets as described herein and increase user comfort and safety. In a ninth aspect, a mobile device, such as a handheld controller, includes a set of sensors for a location tracking system and an electronic rangefinder. This device can be used in methods of interacting with augmented reality systems, including those methods that map a real-world location on a construction site to a virtual world displayed on an augmented reality display. The ninth aspect can provide an improved human-machine interface. In a tenth aspect, a method for preparing three-dimensional building information model data for use in augmented reality applications is provided. This aspect can accelerate the preparation of BIM data for augmented reality views of the site.

[0244] The above aspects can be offered individually or in various combinations. While each aspect has its own advantages, different combinations also provide additional salient or synergistic benefits.

[0245] Referring to the first three aspects, a safety helmet having at least one integrated electronic subsystem can be provided, the safety helmet comprising: an outer portion; and an inner portion, wherein the outer portion and the inner portion are spaced apart within the safety helmet, and wherein the at least one integrated electronic subsystem is mounted between the outer portion and the inner portion. The outer portion and the inner portion may include rigid portions or shells. The outer portion may include a polymer outer shell having a first thickness, and the inner portion may include a carbon fiber inner shell having a second thickness less than the first thickness. The first thickness may be about 1.5 mm, and the second thickness may be about 0.8 mm. The outer portion and the inner portion may be spaced about 20 mm apart for at least half of the circumference of the safety helmet. A first integrated electronic subsystem may be mounted at the rear of the safety helmet, between the outer portion and the inner portion. The first integrated electronic subsystem may include a fan, and the gap between the outer portion and the inner portion allows airflow above the first integrated electronic subsystem. The inner portion may provide one or both of impact protection and penetration protection. The outer portion may be arranged to absorb at least a portion of the energy of an impact. The helmet may include impact-resistant foam disposed between the outer portion and the inner portion. The integrated electronic subsystem may be mounted on the inner portion and / or on the outer portion. The inner portion may include ventilation holes, and the helmet may further include a deformable ventilation coupling for connecting the outer portion and the inner portion, the deformable ventilation coupling allowing air to flow from the ventilation holes to the outside of the outer portion. The deformable ventilation coupling may include a waterproof seal to prevent water from entering the ventilation holes. The deformable ventilation coupling may be attached to the outer portion and the inner portion. The deformable ventilation coupling may include: a first rigid frame for coupling to the inner portion; a second rigid frame for coupling to the outer portion; and a deformable suspension system disposed between the first rigid frame and the second rigid frame. The deformable ventilation coupling may include a rubber component. The deformable ventilation coupling may include two sets of parallel holes, three holes in each set. The outer portion may include a vent aligned with the deformable ventilation coupling during use. The integrated electronic subsystem may include at least one processor and memory. The integrated electronic subsystem may include a computing module for an augmented reality system.

[0246] In one embodiment, a helmet with at least one integrated electronic subsystem may be provided, the helmet comprising: an outer portion; and an inner portion, wherein the outer portion and the inner portion are spaced apart within the helmet, wherein the at least one integrated electronic subsystem is mounted between the outer portion and the inner portion, and wherein impact-resistant foam is constructed between the outer portion and the inner portion. In another embodiment, a deformable ventilated coupling for a helmet with an integrated electronic subsystem may include: a first rigid frame for coupling to the inner portion of the helmet; a second rigid frame for coupling to the outer portion of the helmet; and a deformable suspension system disposed between the first rigid frame and the second rigid frame, the deformable suspension system including perforations to allow air to flow from vents in the inner portion to the exterior of the outer portion, the perforations including waterproof seals. In another case, a helmet with an integrated electronic subsystem may include an external protective portion; an internal detachable portion for mounting the integrated electronic subsystem, the internal detachable portion being worn by the user and including ventilation holes; and a deformable ventilated connector for connecting the external protective portion and the internal detachable portion, the deformable ventilated connector allowing air to flow from the ventilation holes to the outside of the external protective portion.

[0247] Referring to aspects four through seven, a helmet with an integrated electronic subsystem may include multiple battery connection interfaces for connecting multiple removable batteries, wherein the integrated electronic subsystem includes a power subsystem configured to draw power from one of the connected batteries to allow replacement of another of the multiple removable batteries without causing power loss to the integrated electronic subsystem. The helmet may also include multiple removable batteries, wherein the removable batteries include rechargeable batteries. The battery connection interfaces may be laterally mounted within the helmet. Each of the multiple battery connection interfaces may include: a battery socket within the helmet; and a removable housing portion for receiving one of the multiple removable batteries, the removable housing portion being coupled to the helmet around the battery socket to align one removable battery with the battery socket. The integrated electronic subsystem may include a computing module for an augmented reality system, and wherein each removable housing portion forms a wing of a set of glasses for the augmented reality system. The multiple battery connection interfaces may allow removal of at least one of the multiple removable batteries during use on a user's head. The removable housing portion can be removed with one hand by the user. The battery connection ports can be mounted laterally such that when the removable battery is connected to the helmet, the helmet's center of gravity is below its circumferential edge. The plurality of battery connection ports can be symmetrically aligned relative to the front of the helmet, such that the helmet's center of gravity is located on or near its centerline. The plurality of battery connection ports can be mounted laterally such that when the removable battery is connected to the helmet, the helmet's center of gravity is located behind the connected removable battery. One or more of the plurality of removable batteries can further be used to power other peripheral devices used with the helmet.

[0248] In one embodiment, a removable battery housing for a removable battery in a helmet with an integrated electronic subsystem includes: a mechanical interface for coupling with the helmet; and a securing mechanism for securing the removable battery within the removable battery housing when it is not coupled to the helmet, wherein the securing mechanism is arranged to release the removable battery when the removable battery housing is coupled to the helmet via the mechanical interface. The securing mechanism may include a clamping mechanism comprising: a pivoting member; and a force-applying member, wherein when the removable battery housing is not coupled to the helmet, the force-applying member applies a force to a first end of the pivoting member to frictionally secure the removable battery within the removable battery housing, wherein when the removable battery housing is coupled to the helmet, the mechanical interface applies a reaction force to a second end of the pivoting member to move the pivoting member to release the removable battery within the removable battery housing. The removable battery housing may further include a battery biasing member, wherein, when the removable battery housing is attached to the helmet, the battery biasing member applies a force to the removable battery to form an electrical connection between the removable battery and the integrated electronic subsystem of the helmet.

[0249] In another embodiment, a kit for use on a construction site is provided, the kit comprising: a helmet with an integrated augmented reality subsystem; a plurality of removable rechargeable batteries; a set of removable battery housings, each receiving one of the plurality of removable rechargeable batteries in use, at least two of the set of removable battery housings being mechanically coupled to the helmet in use to power the integrated augmented reality subsystem of the helmet; and one or more tracking beacons for determining the location of the helmet within the construction site, each tracking beacon being configured to receive at least one of the plurality of removable rechargeable batteries for power supply in the event of unavailable external power. The kit may also include a charging station for recharging one or more of the plurality of removable rechargeable batteries. The charging station may be arranged to simultaneously recharge more than two of the plurality of removable rechargeable batteries and / or may include a plurality of battery recharging compartments on each side of the charging station. The receiving portion on each side of the charging station may be movable between two positions: an open position for receiving one or more of the plurality of removable rechargeable batteries and a closed position for protecting the terminals of the plurality of battery recharging compartments. The kit may also include a handheld controller. The kit may contain any combination of components as described in the different embodiments herein.

[0250] According to an eighth aspect, a bracket height adjustment mechanism for a safety helmet is provided, the bracket height adjustment mechanism comprising at least: a bracket for positioning the safety helmet on a user's head; and a set of bracket mounting pins, wherein the bracket includes a plurality of spaced-apart holes adjustablely aligned with corresponding holes in a bracket mounting seat receiving the bracket, and wherein the set of bracket mounting pins is removable to select different holes among the plurality of spaced-apart holes to adjust the relative height of the bracket with respect to the bracket mounting seat during use. The bracket mounting pins may include quarter-turn bayonet locking pins. The bracket mounting pins may include foldable handles having a position substantially perpendicular to the face of each mounting pin for rotating the pin. The bracket may include multiple sets of holes, each set containing at least two holes, the holes being at least vertically spaced relative to the safety helmet. In one case, the bracket includes four sets of holes, each set containing two holes, the holes being evenly spaced around the bracket. The height of the bracket can be adjusted within the bracket mounting seat by vertical intervals of 10 mm. The mechanism may also include a bracket mounting base for attaching the bracket to the helmet, the bracket mounting base including a plurality of holes corresponding to the plurality of spaced-apart holes in the bracket of the bracket height adjustment mechanism. Helmets including the bracket height adjustment mechanism, including variations thereof, may be provided. An accompanying method for adjusting the height of a helmet positioned on a user's head may include: rotating a set of bracket mounting pins to remove the pins from a plurality of corresponding holes in the bracket and bracket mounting base of the helmet; selecting a set of alternating mounting holes in at least one of the bracket and the bracket mounting base; moving at least one of the bracket and the bracket mounting base to align with the selected set of alternating mounting holes; re-inserting the bracket mounting pins into the aligned alternating mounting holes; and rotating the set of bracket mounting pins to lock the pins in place.

[0251] As part of a ninth aspect, a method includes: tracking the position and orientation of a mobile device within a construction site; indicating a first point using the mobile device operated by a user wearing a head-mounted display, the first point comprising: a real-world point within the construction site or a virtual point in a virtual space viewed by the user; emitting a directional ranging beam from the mobile device in the direction of the indicated first point; determining a distance to an occupied portion of the space within the construction site using the directional ranging beam; determining the direction of the directional ranging beam; and determining the position of a second point corresponding to the first point using the position and orientation of the mobile device, the direction of the ranging beam, and the distance to the occupied portion of the space, the second point comprising a corresponding virtual point for the real-world point or a corresponding real-world point for the virtual point. The mobile device may include a handheld controller as described herein. The virtual space may be populated using data from a building information model defined relative to a model coordinate system. Tracking may be performed within a tracking coordinate system. The directional ranging beam may be emitted from the mobile device and reflected by an occupied portion of the space, the reflection of which is detected by the mobile device. The method determines the distance to the occupied portion of the space and the direction of the directional ranging beam within the tracking coordinate system; and determines the location of a real-world point within the construction site within the tracking coordinate system. A calibration transformation can be used to determine the correspondence between the tracking coordinate system and the model coordinate system, mapping points between the coordinate systems. Virtual points may include points on surfaces or objects defined as part of the building information model, and the method may include: mapping between the tracking coordinate system and the model coordinate system using a calibrated transformation to determine the corresponding positions of virtual points and real-world points in a common coordinate system; and determining any discrepancies between the indicated corresponding positions of virtual points and real-world points in the common coordinate system. The method may also include indicating the discrepancies between the corresponding positions of virtual points and real-world points in the common coordinate system within a virtual space as viewed by the user. Instructions to match virtual points with real-world points in the common coordinate system can be received from the user; and the method may include updating the positions of surfaces or objects within the building information model.

[0252] In the case of a mobile device including a handheld portable construction tool, the instruction may include: pointing a virtual representation of the handheld portable construction tool to a virtual point of interest; performing ray tracing from a predefined location on the virtual representation of the handheld portable construction tool to a virtual surface or object in the virtual space; and determining the location where the ray traced intersects with the virtual surface or object, the location being presented as the location of the indicated virtual point.

[0253] In the case of a mobile device worn by a user and including a head-mounted display, the instruction may include: pointing a virtual representation of one or more body parts of the user to a point of interest; performing ray tracing from a position defined relative to the virtual representation of the one or more body parts of the user to a virtual surface within the virtual space; and determining the position where the ray traced intersects the virtual surface, the position being presented as the position of the indicated virtual point.

[0254] A directional ranging beam can be launched from a defined location on a mobile device; and the direction of the directional ranging beam can be determined based on the orientation of the mobile device. The directional ranging beam can be launched from the defined location on the mobile device with a configurable directionality, wherein determining the direction of the directional ranging beam includes measuring the configurable directionality at launch. The position and orientation of the mobile device can be provided as a six-DOF attitude within the tracking coordinate system; and the distance to the occupied portion of space and the direction of the directional ranging beam can be used to determine a transformation within the tracking coordinate system that defines the position of a real-world point within the tracking coordinate system.

[0255] The first point may include a virtual point, and the method may include: determining the position of the corresponding real-world point of the virtual point using the position and orientation of the mobile device, the direction of the ranging beam, and the distance to the occupied portion of the space; mapping the real-world point back to the virtual space using a calibrated transformation between the model coordinate system of the virtual space and the coordinate system used for tracking in the real-world space; and displaying the positions of the mapped real-world point and the initially indicated virtual point in the virtual space, including indicating any differences between the mapped real-world point and the virtual point.

[0256] The first point may include a real-world point, and the method may include: indicating the first point by pointing a mobile device at the first point within the construction site; wherein, in this case, determining the location of the corresponding second point includes: determining the location of the first point in a coordinate system for tracking the mobile device within the construction site; mapping the location of the first point to a virtual space to determine the location of the corresponding second point, the corresponding second point including a virtual point within the virtual space; and indicating the location of the corresponding second point within the virtual space to a user via a head-mounted display.

[0257] The method may include: a user selecting a virtual surface or object in a virtual space viewed by the user; determining one or more locations of real-world points corresponding to the selected virtual surface or object using the position and orientation of the mobile device, the direction of the ranging beam, and the distance to an occupied portion of the space; detecting a gesture from the user related to the virtual surface or object; updating the position of the virtual surface or object in the virtual space based on one or more locations of real-world points corresponding to the selected virtual surface or object; and updating the display position of the virtual surface or object in the virtual space viewed by the user. The method may also, or alternatively, include: a user indicating a series of corners forming a portion of an object in the virtual space viewed by the user; determining corresponding locations of real-world points corresponding to the series of corners using the position and orientation of the mobile device, the direction of the ranging beam, and the distance to an occupied portion of the space; mapping the locations of the real-world points to the virtual space; and using the mapped locations to update the locations of the series of corners in the virtual space. The method may include: obtaining a virtual object within a virtual space; using the removable device to indicate multiple real-world points; determining the position of a virtual point corresponding to the multiple real-world points; and aligning the virtual object within the virtual space based on the position of the virtual point. The method may further include: selecting a face of the virtual object; using the virtual point to define a plane within the virtual space; and aligning the face of the virtual object with the plane in the virtual space. The positions of the multiple virtual points may be used to define a work area that sets the rendering distance of the virtual space within a head-mounted display. A first point and a second point may be used to align the virtual object in the virtual space with a physical location within a construction site. The virtual point may include a position defined in the virtual space with reference to the virtual object, wherein the correspondence between real-world points and virtual points can be used to position the virtual object relative to the real-world points. The first point may include a real-world point within the construction site, wherein the corresponding virtual point in the virtual space can be used to set the size of the virtual object within the virtual space. The method may include indicating at least two real-world points within the construction site; determining corresponding virtual points for the two real-world points; and using the distance between the corresponding virtual points in the virtual world to set the size of the virtual object.

[0258] As part of the ninth aspect, a mobile device may exist that interacts with a virtual representation of a construction site viewed by a user through a head-mounted display, the mobile device being detached from the head-mounted display. The mobile device includes: a set of sensors for a position tracking system configured to acquire sensor data to derive one or more of the position and orientation of the mobile device within the construction site; and an electronic rangefinder configured to determine a distance along a line of sight from a known position on the mobile device to an occupied portion of space within the construction site, the occupied portion of the space being remote from the construction tool. The sensor data and the determined distance can be used to determine the position of a point corresponding to the occupied portion of the space, the position being initiated with reference to the position tracking system. The mobile device is configured to be oriented by the user within the construction site to compare a model-defined and measured real-world point within the virtual representation. The mobile device may include a handheld portable construction tool that can be used with the head-mounted display, wherein the head-mounted display includes a set of sensors for a position tracking system configured to acquire sensor data to derive one or more of the position and orientation of the head-mounted display within the construction site. An electronic rangefinder can emit a directional beam to determine distances. The directional beam is emitted from a known location on a mobile device and has a known or measurable emission vector from the known location. The emission vector and the determined distance can be used to determine the three-dimensional position of a point corresponding to an occupied portion of space relative to the known location, and the known location is a known or measurable position in three-dimensional space relative to the position of the mobile device derived from sensor data. The electronic rangefinder can include one or more of the following: ultrasonic ranging devices; and laser ranging devices. The mobile device can include: an orientation sensor for determining the orientation of the mobile device, wherein the orientation from the orientation sensor and the position derived from at least the sensor data from the set of sensors used for the position tracking system can be used to determine the three-dimensional attitude of the mobile device in the coordinate system used for the position tracking system. The mobile device can include an electronic control system to acquire the sensor data and the determined distance, and to determine the position of a point corresponding to an occupied portion of space within the coordinate system of the position tracking system. Alternatively, these control functions can be distributed across one or more electronic devices, including one or more of the following: the mobile device, an integrated electronic subsystem of a helmet, and a remote server. The electronic control system can be configured to: determine the positions of multiple points in an occupied space; obtain data representing the corresponding known positions of the measurement points in a coordinate system used to define the building information model; and use the correspondence between the measured positions and known positions of the multiple points to calculate a transformation to align the coordinate system of the building information model and the location tracking system.

[0259] In some cases, the methods described herein may be provided as computer programs and / or computer program products. In one case, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to: obtain data representing the location of a mobile device from a location tracking system used at a construction site, the location being defined relative to a coordinate system of the location tracking system; obtain data representing the orientation of the mobile device relative to the coordinate system of the location tracking system; obtain data representing the distance from the mobile device to a point in an occupied space within the construction site, the distance being obtained using a ranging beam emitted by the mobile device toward the point, the point being remotely located relative to the mobile device, the mobile device being oriented to indicate the point; obtain data representing the direction of the ranging beam when emitted by the mobile device; and calculate the position of the point, at least within the coordinate system of the location tracking system, by combining the position and orientation of the mobile device, the direction of the ranging beam, and the distance from the mobile device to the point.

[0260] In a tenth aspect, there is a method for preparing three-dimensional building information model (BIM) data for use in augmented reality applications, the method comprising: obtaining planning data defining an activity-based construction schedule, the activity-based construction schedule including multiple tasks to be performed as part of a construction project; obtaining element data representing a set of 3D elements defined within the BIM data; for at least one of the multiple tasks, processing portions of the planning data and the element data associated with the given task to assign a subset of the 3D element set as candidate elements for the given task, the processing including using assignment data configured based on a training set of the planning data having 3D elements assigned to tasks within the planning data; and providing candidate elements for generating a task-specific augmented reality view of a construction site associated with the construction project, the task-specific augmented reality view being associated with the given task. Providing candidate elements may include: displaying a list of candidate elements associated with the given task to a user; receiving from the user a selection of confirmed candidate elements to be used in the task-specific augmented reality view for the given task; and assigning the selection of confirmed candidate elements to the given task. The data defining the selection of confirmed candidate elements and the given task can be used to configure assignment data for further tasks. The method may also include: viewing an augmented reality view of a construction site via a head-mounted display; selecting the given task from a plurality of tasks using an augmented reality user interface displayed within the head-mounted display; and populating the augmented reality view of the construction site with the confirmed candidate elements within a virtual layer of the augmented reality view. The assignment data can be configured based on element name, element type, and one or more element attributes associated with the assigned 3D element.

[0261] According to the eleventh aspect, there exists a method for aligning a building information model with an augmented reality view based on surface matching. The method includes: obtaining an unaligned three-dimensional building information model for an augmented reality view of a construction site, the unaligned three-dimensional building information model being defined in a model coordinate system; for each of a plurality of model surfaces within the three-dimensional building information model: receiving an indication of a model surface among the plurality of model surfaces in the augmented reality view; receiving corresponding measurements of a plurality of locations on corresponding real-world surfaces at the construction site via a mobile handheld device tracked within the construction site, the measurements being defined in a tracking coordinate system; using the measured plurality of locations to determine a plane representing the corresponding real-world surface in the tracking coordinate system; and assigning the plane to the indicated model surface; and calculating a transformation matrix using a corresponding set of the plurality of model surfaces and the assigned plane to align the three-dimensional building information model with the augmented reality view.

[0262] This method allows for rapid and robust alignment of the building information model with a tracking coordinate system, enabling augmented reality views of the construction site (i.e., relevant portions of the building information model are overlaid on the view of the construction site). The method has features such as reference... Figures 16A to 17 The benefits described.

[0263] According to a variation of the eleventh aspect, an augmented reality view is provided within a set of display panels of an augmented reality head-mounted device, which is tracked within a tracking coordinate system. A movable handheld device may include a handheld controller. In this case, receiving measurements of multiple locations corresponding to real-world surfaces at a construction site includes (may include) for each location: determining the orientation of the handheld controller in the tracking coordinate system; measuring the distance to an indicated remote point on the corresponding real-world surface using an electronic rangefinder; and determining the location within the tracking coordinate system using the orientation of the handheld controller, the measured distance, and a known spatial configuration of the handheld controller. Receiving the measurement results of multiple locations corresponding to real-world surfaces at a construction site may additionally or alternatively include: docking the handheld controller to the corresponding real-world surface; determining the orientation of the handheld controller in the tracking coordinate system; and determining each of the multiple locations within the tracking coordinate system using the orientation of the handheld controller and a known spatial configuration of the handheld controller. For example, the handheld controller may be physically placed on a real-world surface.

[0264] According to another variation, the method includes: obtaining spatial definitions of a plurality of model surfaces within a model coordinate system; obtaining spatial definitions of planes corresponding to the real-world surfaces; and calculating a transformation matrix mapping between the spatial definitions, the transformation matrix including rotation, translation, and scaling parameters. The indications of each of the plurality of model surfaces can be constrained such that the model surfaces are orthogonal. Obtaining an misaligned 3D building information model may include using an augmented reality interface to filter out portions of the misaligned 3D building information model before indicating the model surfaces within the augmented reality view.

[0265] Unless otherwise stated, all publications cited herein are incorporated herein by reference. The above embodiments and aspects should be understood as illustrative. Further examples and aspects are contemplated. Although certain components of each example and aspect have been described individually, it should be understood that the functionality described with reference to one example or aspect may be suitably implemented in another example or aspect, and certain components may be omitted depending on the implementation. It should be understood that any feature described with respect to any example or aspect may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other example or aspect, or any combination of any other example or aspect. For example, features described with respect to system components may also be adapted to be performed as part of the described method. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined by the appended claims.

Claims

1. A safety helmet having at least one integrated electronic subsystem, the safety helmet comprising: External part; An internal portion, wherein the external portion and the internal portion are spaced apart within the helmet, and the at least one integrated electronic subsystem is mounted between the external portion and the internal portion; and Multiple battery connection interfaces for connecting multiple removable batteries. The integrated electronic subsystem includes a power subsystem configured to draw power from one of the connected removable batteries among the plurality of removable batteries, enabling the replacement of another removable battery among the plurality of removable batteries without causing power loss to the integrated electronic subsystem. The plurality of battery connection interfaces are laterally mounted within the helmet, and each of the plurality of battery connection interfaces includes: A battery socket, wherein the battery socket is disposed between the inner portion and the outer portion; and A removable housing portion for receiving one of the plurality of removable batteries, the removable housing portion being able to be attached to the helmet around the battery socket to align the one removable battery with the battery socket.

2. The safety helmet according to claim 1, wherein, The removable battery housing includes: A mechanical interface for connection with the safety helmet; and A fixing mechanism is provided for securing the removable battery within the removable battery housing when the removable battery housing is not connected to the safety helmet. The fixing mechanism is arranged to release the removable battery when the removable battery housing is attached to the helmet via the mechanical interface.

3. The safety helmet according to claim 2, wherein, The fixing mechanism includes a clamping mechanism, and the clamping mechanism includes: Pivoting components; and Force-applying components, When the removable battery housing is not connected to the safety helmet, the force-applying member applies force to the first end of the pivoting member to frictionally secure the removable battery within the removable battery housing. When the removable battery housing is connected to the safety helmet, the mechanical interface applies a reaction force to the second end of the pivot member to move the pivot member, thereby releasing the removable battery inside the removable battery housing.

4. The safety helmet according to any one of the preceding claims, wherein, The removable battery housing also includes: Battery biasing components, When the removable battery housing is attached to the helmet, the battery biasing member applies a force to the removable battery to form an electrical connection between the removable battery and the integrated electronic subsystem of the helmet.

5. The safety helmet according to any one of the preceding claims, wherein, The plurality of battery connection interfaces are mounted laterally such that when the removable battery is connected to the helmet, the center of gravity of the helmet is below the circumferential edge of the helmet.

6. The safety helmet according to claim 5, wherein, The plurality of battery connection ports are symmetrically aligned with respect to the front of the helmet, such that the center of gravity of the helmet is located on or near the center line of the helmet.

7. The safety helmet according to claim 5 or 6, wherein, The plurality of battery connection interfaces are mounted laterally such that when the removable battery is connected to the helmet, the center of gravity of the helmet is located at the rear of the connected removable battery.

8. The safety helmet according to any one of claims 1 to 7, wherein, The multiple battery connection interfaces are mounted laterally, such that when the removable battery is connected to the helmet, the helmet's center of gravity is: Located below the circumferential edge of the safety helmet; Located on or near the center line of the safety helmet; as well as Located at the rear of the attached removable battery.

9. The safety helmet according to any one of claims 1 to 8, wherein, One or more of the removable batteries can also be used to power other peripheral devices used with the helmet.

10. The safety helmet according to any one of the preceding claims, wherein, The plurality of battery connection interfaces allow at least one of the plurality of removable batteries to be removed during use on the user's head.

11. The safety helmet according to claim 10, wherein, The detachable housing portion can be removed by the user with one hand.

12. The safety helmet according to any one of the preceding claims, wherein, The integrated electronic subsystem includes a memory and at least one processor.

13. The safety helmet according to claim 12, wherein, The integrated electronic subsystem includes a computing module for the augmented reality system.

14. The safety helmet according to any one of the preceding claims, wherein, Each detachable housing section forms a wing for the viewing component of the augmented reality system.

15. The safety helmet according to any one of the preceding claims, wherein, The outer portion includes a polymer shell with a first thickness, and the inner portion includes a carbon fiber inner shell with a second thickness, the second thickness being less than the first thickness.

16. The safety helmet according to claim 15, wherein, The first thickness is about 1.5 mm, the second thickness is about 0.8 mm, and wherein, for at least half of the circumference of the helmet, the outer portion and the inner portion are spaced about 20 mm apart.

17. The safety helmet according to any one of the preceding claims, wherein, A first integrated electronic subsystem is mounted on one or more of the inner and outer portions at the rear of the helmet, between the outer portion and the inner portion.

18. The safety helmet according to any one of the preceding claims, wherein, The first integrated electronic subsystem includes a fan, and the gap between the outer portion and the inner portion allows air to flow over the first integrated electronic subsystem.

19. The safety helmet according to any one of the preceding claims, wherein, The internal portion provides one or more of impact protection and penetration protection.

20. The safety helmet according to any one of the preceding claims, further comprising: Impact-resistant foam, wherein the impact-resistant foam is disposed between the outer portion and the inner portion.

21. The safety helmet according to any one of the preceding claims, wherein: The internal portion includes ventilation holes, and The safety helmet also includes: A deformable ventilation connector for connecting the outer portion and the inner portion, the deformable ventilation connector allowing air to flow from the ventilation holes to the outside of the outer portion.

22. The safety helmet according to claim 21, wherein, The deformable ventilation connector includes a waterproof seal to prevent water from entering the ventilation hole.

23. The safety helmet according to claim 21 or claim 22, wherein, The deformable ventilation connector is attached to the outer portion and the inner portion.

24. The safety helmet according to any one of claims 21 to 23, wherein, The deformable ventilation connector includes: A first rigid frame, the first rigid frame being used for connection to the internal portion; A second rigid frame, the second rigid frame being used for connection to the external portion; and A deformable suspension system, wherein the deformable suspension system is arranged between the first rigid frame and the second rigid frame.

25. The safety helmet according to any one of claims 21 to 24, wherein, The deformable ventilation connector includes a rubber component.

26. The safety helmet according to any one of claims 21 to 25, wherein, The external portion includes a vent that aligns with the deformable ventilation connector during use.

27. The safety helmet according to any one of claims 1 to 26, wherein the safety helmet comprises: Bracket mounting base; A bracket for positioning the helmet on the user's head; as well as A set of bracket mounting pins, The bracket includes a plurality of spaced-apart holes, which can be aligned with corresponding holes in the bracket mounting base that receives the bracket by adjustment. The set of bracket mounting pins is removable to select different holes among the plurality of spaced holes to adjust the relative height of the bracket with respect to the bracket mounting base during use.

28. The safety helmet according to claim 27, wherein, The set of bracket mounting pins includes a quarter-turn bayonet locking pin.

29. The safety helmet according to claim 28, wherein, The bracket mounting pin includes a foldable handle, which has a position substantially perpendicular to the face of each mounting pin for rotating the pin.

30. A safety helmet with an integrated electronic subsystem, the safety helmet comprising: Multiple battery connection interfaces for connecting multiple removable batteries. The integrated electronic subsystem includes a computing module for the augmented reality system. The integrated electronic subsystem includes a power subsystem configured to draw power from one of the connected removable batteries, enabling the replacement of another removable battery without power loss to the integrated electronic subsystem. Each of the plurality of battery connection interfaces includes a battery socket and is configured to connect to a removable housing portion, the removable housing portion being configured to receive one of the plurality of removable batteries. The removable housing portion is connectable to the helmet to align the one removable battery with the battery socket. The plurality of battery connection interfaces are mounted laterally on opposite sides of the helmet, and each detachable housing portion forms a wing for the viewing component of the augmented reality system when connected.

31. A kit for use on a construction site, the kit comprising: A safety helmet with an integrated augmented reality subsystem; Multiple removable, rechargeable batteries; A set of removable battery housings, each receiving one of the plurality of removable rechargeable batteries in use, wherein at least two of the removable battery housings in the set are mechanically coupled in use to a laterally arranged battery connection interface located on a corresponding side of the helmet to power the integrated augmented reality subsystem of the helmet; as well as One or more tracking beacons for determining the location of the safety helmet within the construction site, each tracking beacon being configured to receive at least one of the plurality of removable rechargeable batteries for power supply in the event that external power is unavailable.

32. The kit of claim 31, further comprising: A charging station that recharges one or more of the plurality of removable rechargeable batteries.

33. The kit according to claim 32, wherein, The charging station is arranged to simultaneously recharge more than two of the plurality of removable rechargeable batteries.

34. The kit according to claim 33, wherein, The charging station includes multiple battery recharging compartments on each side of the charging station.

35. The kit according to claim 34, wherein, The receiving portion on each side of the charging station is movable between two positions: an open position for receiving one or more of the plurality of removable rechargeable batteries; And a closed position for protecting the terminals of the plurality of battery recharge compartments.

36. The kit according to any one of claims 31 to 35, the kit further comprising: Handheld controller.

37. A deformable ventilated connector for a safety helmet, the safety helmet having an integrated electronic subsystem, the deformable ventilated connector comprising: A first rigid frame, the first rigid frame being used to connect to the internal portion of the helmet; A second rigid frame is used to connect to the outer portion of the helmet; as well as A deformable suspension system disposed between a first rigid frame and a second rigid frame, the deformable suspension system including holes to allow air to flow from ventilation holes in the inner portion to the outside of the outer portion, the holes including waterproof seals.

38. A bracket height adjustment mechanism for a safety helmet, the bracket height adjustment mechanism comprising: A bracket for positioning the helmet on the user's head; as well as A set of bracket mounting pins, said set of bracket mounting pins including a quarter-turn bayonet locking pin. The bracket includes a plurality of spaced-apart holes, which can be aligned with corresponding holes in a bracket mounting base that receives the bracket by adjustment. The set of bracket mounting pins is removable to select different holes among the plurality of spaced holes, thereby adjusting the relative height of the bracket with respect to the bracket mounting base during use.

39. The mechanism according to claim 38, wherein, The bracket mounting pin includes a foldable handle, which has a position substantially perpendicular to the face of each mounting pin for rotating the pin.

40. The mechanism according to any one of claims 38 to 39, wherein, The bracket includes multiple sets of holes, each set including at least two holes, which are at least vertically spaced apart with respect to the safety helmet.

41. The mechanism according to claim 40, wherein, The bracket includes four sets of holes evenly spaced around the bracket, each set of holes including two holes.

42. The mechanism according to any one of claims 38 to 41, wherein, The height of the bracket can be adjusted vertically by up to 10mm within the bracket mounting base.

43. The mechanism according to any one of claims 38 to 42, further comprising: A bracket mounting base for connecting the bracket to the safety helmet, the bracket mounting base including a plurality of holes corresponding to the plurality of spaced-apart holes in the bracket of the bracket height adjustment mechanism.

44. A safety helmet comprising a bracket height adjustment mechanism according to any one of claims 38 to 43.

45. A method for adjusting the height of a safety helmet, the method comprising: Rotate a set of quarter-turn bayonet locking pins to remove the pins from multiple sets of corresponding holes in the bracket and bracket mount of the helmet; Select at least one set of alternating mounting holes from the bracket and the bracket mounting base; Move at least one of the bracket and the bracket mounting base to align with a selected set of alternating mounting holes; Reinsert the quarter-turn bayonet locking pin into the aligned alternating mounting hole; as well as Rotate the set of quarter-turn bayonet locking pins to lock the pins in place.

46. ​​A handheld controller for interacting with a virtual representation of a construction site viewed by a user through a head-mounted display, the handheld controller being detached from the head-mounted display, the handheld controller comprising: A set of sensors for a location tracking system, the set of sensors being configured to acquire sensor data to derive one or more of the position and orientation of the handheld controller within the construction site; as well as An electronic rangefinder is configured to determine the distance from a known location on the handheld controller, along a line of sight, to an occupied portion of space within the construction site, the occupied portion of space being remote from the handheld controller. The sensor data and the determined distance can be used to determine the location of a point corresponding to the occupied portion of the space, the location being defined with reference to the location tracking system. The handheld controller is configured for the user to orient itself within the construction site in order to compare points defined in the model with measured real-world points within the virtual representation.

47. A kit for use on a construction site, the kit comprising: The safety helmet according to any one of claims 1 to 30; Multiple removable, rechargeable batteries; A set of removable battery housings, each receiving one of the plurality of removable rechargeable batteries in use, wherein at least two of the removable battery housings in the set are mechanically coupled to the helmet in use to power the helmet's integrated augmented reality subsystem; One or more tracking beacons for determining the location of the safety helmet within the construction site, each tracking beacon being configured to receive at least one of the plurality of removable rechargeable batteries for power supply in the event that external power is unavailable; The handheld controller of claim 46 is configured to receive at least one of the plurality of removable rechargeable batteries for power supply; as well as A charging station for recharging one or more of the plurality of removable rechargeable batteries.

48. A method, the method comprising: Track the location and orientation of mobile devices within construction sites; The first point is indicated using the mobile device operated by a user wearing a head-mounted display, the first point comprising: The real-world points within the construction site, or Virtual points within the virtual space as seen by the user; A directional ranging beam is emitted from the mobile device in the direction of the indicated first point; Using the directional ranging beam, the distance to the occupied portion of the space within the construction site is determined; Determine the direction of the directional ranging beam; Using the location and orientation of the mobile device, the direction of the ranging beam, and the distance to the occupied portion of the space, the location of a second point corresponding to the first point is determined, the second point including a virtual point corresponding to the real-world point or a real-world point corresponding to the virtual point.

49. The method according to claim 48, wherein, The virtual space is populated using data from a building information model defined with respect to the model coordinate system.

50. The method according to claim 49, wherein: The tracking is performed within the tracking coordinate system; The directional ranging beam is emitted from the mobile device and reflected by the occupied portion of the space, and the reflection of the directional ranging beam is detected by the mobile device. The distance to the occupied portion of the space and the direction of the directional ranging beam are determined within the tracking coordinate system; as well as The location of the real-world point within the construction site is determined within the tracking coordinate system.

51. The method according to claim 50, wherein, A calibrated transformation is used to determine the correspondence between the tracking coordinate system and the model coordinate system, the calibrated transformation mapping points between the coordinate systems.

52. The method according to claim 51, wherein, The virtual points include points on surfaces or objects that are defined as part of the building information model, and the method includes: The calibration transformation is used to map between the tracking coordinate system and the model coordinate system to determine the corresponding positions of the virtual point and the real-world point in a common coordinate system; and Determine any discrepancies between the indicated virtual point and the corresponding real-world point in the common coordinate system.

53. The method according to claim 52, wherein the method comprises: It indicates the difference between the corresponding positions of the virtual point and the real-world point in the common coordinate system within the virtual space as seen by the user.

54. The method according to claim 52 or 53, wherein the method comprises: The user receives an instruction to match the virtual point with the real-world point in the public coordinate system. as well as Update the position of the surface or object within the building information model.

55. The method according to any one of claims 48 to 54, wherein: The mobile device includes handheld portable construction tools; The instructions include: The virtual representation of the handheld portable construction tool is pointed to a virtual point of interest; Ray tracing is performed from a predefined position on the virtual representation of the handheld portable construction tool to a virtual surface or object within the virtual space; and The location where the ray from the ray tracing intersects the virtual surface or object is determined, and the location is presented as the position of the indicated virtual point.

56. The method according to any one of claims 48 to 55, wherein: The mobile device is worn by the user and includes the head-mounted display; The instructions include: Point virtual representations of one or more body parts of the user to points of interest; Ray tracing is performed from the position defined by the virtual representation relative to one or more body parts of the user to a virtual surface within the virtual space; and The location where the ray from the ray tracing intersects the virtual surface is determined, and the location is presented as the position of the indicated virtual point.

57. The method according to any one of claims 48 to 56, wherein: The directional ranging beam is emitted from a defined position on the mobile device; and The direction of the directional ranging beam is determined based on the orientation of the mobile device.

58. The method according to any one of claims 48 to 57, wherein, The directional ranging beam is emitted from a defined position on the mobile device with a configurable directionality, wherein determining the direction of the directional ranging beam includes measuring the configurable directionality at the time of emission.

59. The method according to any one of claims 48 to 58, wherein: The position and orientation of the mobile device are provided as a six-degree-of-freedom (6FOD) pose within the tracking coordinate system; as well as The distance to the occupied portion of the space and the direction of the directional ranging beam are used to determine the transformation within the tracking coordinate system, which defines the position of the real-world point within the tracking coordinate system.

60. The method according to any one of claims 48 to 59, wherein, The first point includes a virtual point, and the method includes: The location of the virtual point corresponding to the real-world point is determined using the location and orientation of the mobile device, the direction of the ranging beam, and the distance to the occupied portion of the space. Using a transformation calibrated between the model coordinate system of the virtual space and the coordinate system used for tracking in the real-world space, the real-world points are mapped back to the virtual space; and Displays the positions of real-world points mapped in the virtual space and the positions of initially indicated virtual points, including any differences between the mapped real-world points and the virtual points.

61. The method according to any one of claims 48 to 59, wherein, The first point includes a real-world point, and the method includes: The first point is indicated by pointing the mobile device at the first point within the construction site; and Determining the location of the corresponding second point includes: Determine the position of the first point in the coordinate system used to track the mobile device within the construction site; Mapping the position of the first point to the virtual space to determine the position of the corresponding second point, wherein the corresponding second point includes virtual points within the virtual space; and The user is shown the location of the corresponding second point within the virtual space via the head-mounted display.

62. The method according to any one of claims 48 to 61, the method comprising: The user selects a virtual surface or object in the virtual space that the user is viewing; Using the location and orientation of the mobile device, the direction of the ranging beam, and the distance to the occupied portion of the space, determine one or more locations of real-world points corresponding to the selected virtual surface or object; Detect gestures from the user related to the virtual surface or object; Update the position of the virtual surface or object in the virtual space based on one or more positions of real-world points corresponding to the selected virtual surface or object; as well as Update the display position of the virtual surface or object in the virtual space as seen by the user.

63. The method according to any one of claims 48 to 62, the method comprising: A series of corners that form part of an object in the virtual space as seen by the user, as indicated by the user; Using the location and orientation of the mobile device, the direction of the ranging beam, and the distance to the occupied portion of the space, the corresponding positions of real-world points corresponding to the series of corners are determined; Map the location of a real-world point to the virtual space; as well as Update the positions of the series of corners in the virtual space using the mapped positions.

64. The method according to any one of claims 48 to 63, the method comprising: Obtain the virtual objects within the virtual space; Use the mobile device to indicate multiple real-world points; Determine the positions of the virtual points corresponding to the plurality of real-world points; as well as Based on the position of the virtual point, align the virtual object within the virtual space.

65. The method according to claim 64, further comprising: Select the face of the virtual object; The virtual points are used to define planes within the virtual space; as well as Align the face of the virtual object with the plane in the virtual space.

66. The method according to any one of claims 48 to 65, wherein, The method is used to measure multiple locations on the corresponding real-world surface of the construction site, and the method further includes: Using the measured multiple locations, a plane representing the corresponding real-world surface within the tracking coordinate system is determined; Indicates the corresponding model surface within the 3D building information model; Assign the plane to the indicated model surface; and Using at least the indicated model surface and the corresponding assigned plane, a transformation matrix is ​​calculated to align the 3D building information model with the augmented reality view.

67. The method according to any one of claims 48 to 66, wherein, The positions of multiple virtual points are used to define a working area, which sets the rendering distance for the virtual space within the head-mounted display.

68. The method according to any one of claims 48 to 67, wherein, The first point and the second point are used to align the virtual objects in the virtual space with their physical locations within the construction site.

69. The method according to claim 68, wherein, The virtual point includes a location in the virtual space defined with reference to the virtual object, and wherein the correspondence between the real-world point and the virtual point is used to locate the virtual object relative to the real-world point.

70. The method according to any one of claims 48 to 69, wherein, The first point includes a real-world point within the construction site, and the corresponding virtual point in the virtual space is used to set the size of the virtual object within the virtual space.

71. The method according to claim 70, further comprising: Indicate at least two real-world points within the construction site; Determine the corresponding virtual point for the two real-world points; as well as The size of the virtual object is set by using the distance between corresponding virtual points in the virtual world.

72. A mobile device for interacting with a virtual representation of a construction site viewed by a user through a head-mounted display, the mobile device being detached from the head-mounted display, the mobile device comprising: A set of sensors for a location tracking system, the set of sensors being configured to acquire sensor data to derive one or more of the location and orientation of the mobile device within the construction site; as well as An electronic rangefinder is configured to determine the distance from a known location on the mobile device along a line of sight to an occupied portion of space within the construction site, the occupied portion of space being remote from the construction equipment. The sensor data and the determined distance can be used to determine the location of a point corresponding to the occupied portion of the space, the location being defined with reference to the location tracking system. The mobile device is configured to be oriented by the user within the construction site to compare points defined in the model with measured real-world points within the virtual representation.

73. The mobile device according to claim 72, wherein, The mobile device includes a handheld portable construction tool that can be used with the head-mounted display, wherein the head-mounted display includes a set of sensors for the position tracking system, the set of sensors being configured to acquire sensor data to derive one or more of the position and orientation of the head-mounted display within the construction site.

74. The mobile device according to any one of claims 72 to 73, wherein, The electronic rangefinder emits a directional beam to determine the distance. The directional beam is emitted from a known location on the mobile device and has a known or measurable emission vector originating from the known location. Wherein, the emission vector and the determined distance can be used to determine the three-dimensional position of the point corresponding to the occupied portion of the space relative to the known position, and Wherein, the known location is a known or measurable location in three-dimensional space relative to the location of the mobile device derived from the sensor data.

75. The mobile device according to any one of claims 72 to 74, wherein, The electronic rangefinder includes one or more of the following: Ultrasonic ranging equipment; and Laser ranging equipment.

76. The mobile device according to any one of claims 72 to 75, wherein the mobile device comprises: An orientation sensor, used to determine the orientation of the mobile device. The orientation from the orientation sensor and the position derived from at least the sensor data from the set of sensors used in the position tracking system are used to determine the three-dimensional attitude of the mobile device in the coordinate system used in the position tracking system.

77. The mobile device according to any one of claims 72 to 76, wherein the mobile device comprises: An electronic control system is configured to acquire the sensor data and the determined distance, and determine the position of the point corresponding to the occupied portion of the space within the coordinate system of the position tracking system.

78. The mobile device according to claim 77, wherein, The electronic control system is configured as follows: Determine the locations of multiple points within the occupied space; Obtain data representing the known location of the measured point within the coordinate system used to define the building information model; as well as The transformation is calculated using the correspondence between the measured positions of the plurality of points and the known positions, in order to align the coordinate systems of the building information model and the location tracking system.

79. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Data representing the location of a mobile device is obtained from a location tracking system used at a construction site, the location being defined with respect to the coordinate system of the location tracking system; Obtain data representing the orientation of the mobile device relative to the coordinate system of the position tracking system; Data representing the distance from the mobile device to a point in an occupied space within the construction site is obtained using a ranging beam emitted by the mobile device toward the point, which is remotely located relative to the mobile device, which is oriented to indicate the point. Obtain data indicating the direction of the ranging beam when it is emitted by the mobile device; as well as The position of the point is calculated, at least within the coordinate system of the position tracking system, by combining the position and orientation of the mobile device, the direction of the ranging beam, and the distance from the mobile device to the point.

80. A method comprising: Obtain an misaligned 3D building information model for an augmented reality view of a construction site, wherein the misaligned 3D building information model is defined within a model coordinate system; For each of the multiple model surfaces within the three-dimensional building information model: Receive an indication of a model surface among the plurality of model surfaces within the augmented reality view; The measurement results are received by a mobile handheld device within the construction site, which is defined within the tracking coordinate system, at multiple locations on the real-world surface of the construction site. Using the measured multiple locations, a plane representing the corresponding real-world surface within the tracking coordinate system is determined; as well as Assign the plane to the indicated model surface; as well as Using the multiple model surfaces and a corresponding set of assigned planes, a transformation matrix is ​​calculated to align the 3D building information model with the augmented reality view.

81. The method according to claim 80, wherein, The augmented reality view is provided within a set of display panels of an augmented reality head-mounted device, which is tracked within the tracking coordinate system.

82. The method according to claim 81, wherein, The portable handheld device includes a handheld controller.

83. The method according to any one of claims 80 to 82, wherein, The portable handheld device includes a portable computing device with a lidar sensor.

84. The method according to any one of claims 80 to 83, wherein, Receiving measurement results from multiple locations on the corresponding real-world surface of the construction site includes, for each location: Determine the attitude of the movable handheld device in the tracking coordinate system; Using an electronic ranging device, the distance to the indicated remote point on the corresponding real-world surface is measured; as well as The position within the tracking coordinate system is determined using the posture of the mobile handheld device, the measured distance, and the known spatial configuration of the mobile handheld device.

85. The method according to claim 82, wherein, Receiving measurement results from multiple locations on the corresponding real-world surface of the construction site includes: Connect the handheld controller to the corresponding real-world surface; Determine the attitude of the handheld controller in the tracking coordinate system; and Using the pose of the handheld controller and the known spatial configuration of the handheld controller, each of the plurality of positions within the tracking coordinate system is determined.

86. The method according to claim 84 or claim 85, wherein the method comprises: Obtain the spatial definition of the plurality of model surfaces within the model coordinate system; Obtain the spatial definition of the plane corresponding to the real-world surface; as well as Calculate the transformation matrix that maps between the spatial definitions, the transformation matrix including rotation parameters, translation parameters, and scaling parameters.

87. The method according to any one of claims 80 to 86, wherein, The indications of each of the plurality of model surfaces are constrained such that the model surfaces are orthogonal.

88. The method according to any one of claims 80 to 87, wherein, Obtaining misaligned 3D building information models includes: Before directing the model surface within the augmented reality view, the augmented reality interface is used to filter out portions of the misaligned 3D building information model.

89. A method for preparing three-dimensional 3D Building Information Model (BIM) data for use in augmented reality applications, the method comprising: Obtain planning data to define an activity-based construction plan, which includes multiple tasks to be performed as part of a construction project; Obtain element data representing the set of 3D elements defined within the BIM data; For at least one of the plurality of tasks, the element data and the portion of the plan data associated with the given task are processed to assign a subset of the 3D element set as candidate elements for the given task. The processing includes using assignment data configured using a training set based on the plan data, wherein 3D elements are assigned to tasks within the plan data. as well as The candidate elements are provided to generate a task-specific augmented reality view of the construction site associated with the construction project, the task-specific augmented reality view being associated with the given task.

90. The method according to claim 89, wherein, The candidate elements provided include: Display a list of candidate elements associated with the given task to the user; Receive from the user a selection of candidate elements to be used in a task-specific augmented reality view for the given task; and The selection of the confirmed candidate elements is assigned to the given task.

91. The method according to claim 90, wherein, The selection of the confirmed candidate elements and the data for the given task are defined to configure the assignment data for use in other tasks.

92. The method according to claim 90 or claim 91, wherein the method comprises: View an augmented reality view of the construction site via a head-mounted display; Using the augmented reality user interface displayed within the head-mounted display, select the given task from the plurality of tasks; as well as The augmented reality view of the construction site is populated with the confirmed candidate elements within the virtual layer of the augmented reality view.

93. The method according to any one of claims 90 to 92, wherein, The assignment data is configured based on one or more of the element name, element type, and one or more element attributes associated with the assigned 3D element.

Images