Aligning and augmenting portions of a physical infrastructure with at least one information element

Abstract

The present disclosure relates to a computer-implemented method for augmenting a physical infrastructure with at least one information element (4), comprising: - establishing a first spatial environment (1) within the physical infrastructure, - defining at least one second spatial environment (2) as a partial subspace of the first spatial environment (1), - identifying at least one first marker identifier and at least one second marker identifier, the at least one first marker identifier corresponding to the at least one first marker item (5) and the at least one second marker identifier corresponding to the at least one second marker item (5), - defining the at least one information element (4) in relation to at least one component of the physical infrastructure, - augmenting the physical infrastructure with the at least one information element (4).

Description

Technical Field

[0001] The embodiments disclosed herein relate to a computer-implemented method for enhancing physical infrastructure using at least one information element. The embodiments disclosed herein also relate to augmented reality engines, mobile devices, computer program products, and computer-readable storage media corresponding to the disclosed method. Background Technology

[0002] Augmented Reality (AR) provides users with interactive experiences in real-world environments. Objects residing in the real-world environment are augmented with computer-generated information. The overlay information displayed can be interwoven with the physical reality in augmented reality, making it perceived by the user as an immersive scene of a real environment. Augmented Reality can be used to enhance natural environments or situations and provide users or operators with a perceptually rich experience. With the help of advanced augmented reality technologies, environmental information about the real world around a user can become interactive and can be manipulated by the user. In augmented reality, information about the environment and its objects is overlaid on the real world. The displayed information can be virtual or real, for example, allowing the perception of other real-world sensed or measured information, such as electromagnetic radio waves overlaid with their precise alignment with their spatial positions. Augmentation techniques are typically performed in real time and within a semantic context with environmental elements or objects.

[0003] In many use cases, it is necessary to place augmented reality annotations relative to specific locations or objects in the physical world. Other users may later want to retrieve, view, or edit this information when they are near the corresponding location or object.

[0004] In industrial applications, this is particularly useful for information related to a piece of physical infrastructure. However, the applications are not limited to industry. A specific example could be information related to machine commissioning, servicing, and maintenance, such as the type of materials / parameters, which can be provided in advance and / or permanently annotated during commissioning, servicing, and maintenance activities.

[0005] Another example is a construction site, which can be "digitally built" in real time at a later location during the design process by combining 3D models and information with georeferenced data. This enables improved on-site design and planning discussions, installation verification, clash detection, and increased efficiency during construction / installation.

[0006] Another example is live data feeds from machines. Data can be made available anytime, anywhere, in any desired form via AR.

[0007] Another example is using a head-mounted augmented reality display to view step-by-step workflow information for equipment used to automate repair processes in chemical or pharmaceutical plants.

[0008] Another example is pre-inspection scheduling in power plants, which uses augmented reality apps on smartphones to create annotations to document potential problems.

[0009] There are many different methods for creating and subsequently retrieving augmented reality (AR) content. These common methods include tag-based AR, where AR content is created in a 3D graphics programming environment and anchored to 2D visual tags. The AR content can then be retrieved when the 2D visual tag is within the camera of a client / mobile device manipulated by the user. Tag-based AR is often used in AR marketing applications, such as placing 3D models on top of magazine advertisements.

[0010] Another common approach is object-based augmented reality. Augmented reality content is created in a 3D graphics programming environment and then anchored to a 3D computer-aided design (CAD) data model. The augmented reality content is retrieved when the client device / mobile device detects a real-world object using a model-based tracking method. Object-based augmented reality is frequently used in maintenance applications in industrial environments.

[0011] Another common approach is georeferenced augmented reality. Augmented reality content is generated and then retrieved from georeferenced content. This is limited by the accuracy of the GPS system used.

[0012] Another conventional approach involves placing a hologram, essentially a 3D model, within an augmented environment that the user's client device can recognize. Then, when another user uses the same client device at the same location, the hololen can identify that location based on a 3D reconstruction of the environment and display the hologram at that location.

[0013] Another conventional approach is wide-area RF positioning technology, such as ultra-wideband (UWB), which can track the location of augmented reality devices with an accuracy of a few centimeters. This requires either installing base stations for tracking or measuring the signal strength of the built-in wireless network on the installed field device.

[0014] Another conventional approach is marker-based optical tracking. Cameras on the augmented reality device (AR) identify optical markers placed at various locations throughout the factory. Each marker is designed to be easily recognized by image processing techniques. Software on the AR device detects the markers, thus identifying which part of the factory the user is in; then, pose estimation is performed based on the viewpoint from which the marker appears in the device's camera position. Therefore, when a marker is detected, the AR device has a good estimate of its precise location within the factory. This technique has the drawback that specific markers must be placed in precisely defined locations within the factory.

[0015] Another conventional approach is wide-area visual-inertial ranging. For this, an augmented reality device with a camera and additional inertial / magnetic sensors is first used to map the area of ​​the facility. Data is collected to create a (typically sparse) 3D feature map or point cloud of the environment. This can be stored on the AR device itself or on a server. The AR device then uses optical features detected in its camera images to search the created feature map and thus infer its current location. This technique has the drawback that variations in brightness (e.g., due to solar radiation) interfere with the quality of feature recognition during localization.

[0016] Another conventional approach involves laser scanning, photogrammetry, and subsequent server-side image processing. Here, as a first step, a user creates a laser scan of the location, for example, by pushing a cart equipped with a laser scanner and an omnidirectional camera through a building and pressing a "scan" button every few meters. This creates a colored point cloud, which is uploaded to the server. Subsequently, to localize the AR device, the AR device takes one or more photos of its location and sends them to the server. The server performs image processing to determine where the image might be within the stored point cloud and returns it as a pose estimate to the AR device.

[0017] Each of the technologies listed above suffers from some drawbacks, such as in preparation, long-term durability of the markers, spatial accuracy, or interoperability across these technologies. Summary of the Invention

[0018] Therefore, there is a need for a method and apparatus for precisely enhancing physical infrastructure at an accurate location relative to the actual device.

[0019] The embodiments described herein generally relate to computer-implemented methods for enhancing physical infrastructure with at least one information element.

[0020] In one embodiment, the method includes computer-implemented steps, the steps comprising: - Obtain 3D data of a first spatial environment within the physical infrastructure, the size of which is determined by a coordinate system of the first spatial environment. - At least one second spatial environment is defined as a partial subspace of the first spatial environment, the size of the at least one second spatial environment is determined by the coordinate system of the at least one second spatial environment, and the at least one second spatial environment includes at least one first marker item and at least one second marker item. - Load or create at least one simultaneous localization and mapping (SLAM) map of the at least one second spatial environment (3), wherein the size of the at least one SLAM map is determined by the coordinate system of the at least one SLAM map. Align the coordinate system of the at least one simultaneous positioning and mapping map with the coordinate system of the at least one second spatial environment. - Identify at least one first tag identifier and at least one second tag identifier, wherein the at least one first tag identifier corresponds to the at least one first tag item, and the at least one second tag identifier corresponds to the at least one second tag item. - By aligning the coordinate system of the first spatial environment with the coordinate system of the at least one second spatial environment using the at least one first marker and the at least one second marker, the absolute position and orientation of the at least one second spatial environment relative to the first spatial environment can be determined. - Define at least one information element associated with at least one component of the physical infrastructure. - Determine the spatial coordinates of the information element in the coordinate system of the at least one second spatial environment. - Enhance the physical infrastructure using at least one information element at the determined spatial coordinates.

[0021] According to an embodiment, a method is provided for defining and aligning a portion of a physical infrastructure subspace using markers, wherein the portion of the subspace can be enhanced with information elements at determined spatial coordinates.

[0022] The first spatial environment represents at least a portion of the physical infrastructure. In the case of the first spatial environment, a coordinate system (the coordinate system of the first spatial environment) is introduced within the physical infrastructure. The coordinate system of the first spatial environment has the advantage that it can be used to align at least one second spatial environment within the physical infrastructure.

[0023] In an advantageous embodiment that includes at least one second spatial environment, the multiple second spatial environments may overlap.

[0024] The at least one Simultaneous Localization and Mapping (SLAM) map is either newly created or at least one Simultaneous Localization and Mapping (SLAM) map is loaded.

[0025] To create at least one Simultaneous Localization and Mapping (SLAM) map, the SLAM map is newly created using an arbitrary coordinate system (the coordinate system of the SLAM map). The SLAM map contains at least two SLAM markers, each having at least one SLAM marker identifier within the coordinate system of the SLAM map.

[0026] Identify the marker identifier. Then, align at least one coordinate system that simultaneously locates and maps the map with at least one coordinate system of a second spatial environment.

[0027] To load at least one Simultaneous Localization and Mapping Map (SLAM map), the coordinate system of at least one SLAM map is aligned with the coordinate system of at least one second spatial environment by using SLAM markers with SLAM marker identifiers and augmented reality algorithms.

[0028] According to one embodiment, physical infrastructure includes technical infrastructure, in-building infrastructure, outdoor infrastructure, production plant infrastructure, and / or factory infrastructure. Physical infrastructure can also be "physical" in virtual reality applications and refers to physical infrastructure, albeit virtual. According to another embodiment, physical infrastructure is three-dimensional.

[0029] According to one embodiment, at least one component of the physical infrastructure is a technical component.

[0030] The server can manage this subspace and its associated metadata, and can provide image processing capabilities such as image recognition, OCR, or 3D object detection.

[0031] The information elements provided by the method according to the embodiments can assist users in performing actions at accurate locations and increase the accuracy of those actions.

[0032] The subspace or at least one second spatial environment may also be referred to as an augmented reality bubble, which is understood below as a spatial region with defined constraints and forming an internal dimensional system. While the bubble can be implemented from any arbitrary geometry, a spherical size may be chosen for simplicity. However, the term "bubble" is not limited to a sphere. Alternatively, the straight geometry according to alternative embodiments may include straight edges to support modular stacking of bubbles within a mathematically hyperordinate coordinate system. According to an embodiment, the spherical size of the bubble may be chosen to be 10 meters in diameter, enclosing a specific physical location, such as a specific GPS location outdoors, or a specific conference room or specific machine indoors.

[0033] Examples could be substation buildings used to supply power to railways (or any other infrastructure), poles installed along railway tracks (or their future locations), gas turbines within a gas-fired power plant, pumping stations for oil and gas transportation, and many more. Specifically, a bubble can also refer to a section of larger physical infrastructure, such as a segment of a chemical plant.

[0034] Each bubble has a coordinate system. Typically, this has its origin inside the bubble, and one of its axes points upwards. The units for the bubble coordinate system are usually fixed, such as meters.

[0035] For at least one first marker and at least one second marker, which may also be referred to as an identifier or anchor point, it is understood below as a fixed point for registering and aligning the bubble, which is also referred to as a partial subspace in at least one second spatial environment or physical space, such as the physical infrastructure.

[0036] Each bubble contains a set of at least two anchor points, which helps to accurately position the bubble within the physical world. Each anchor point has a fixed position in the bubble coordinate system, such as the coordinate system of at least one second spatial environment.

[0037] Anchor points (at least one first marker and at least one second marker identifier) ​​are used to determine the absolute position and orientation of at least one second spatial environment relative to the first spatial environment by aligning the coordinate system of the first spatial environment and the coordinate system of at least one second spatial environment.

[0038] Each anchor has at least one anchor identifier (at least one first tag identifier corresponding to at least one first tag item and at least one second tag identifier corresponding to at least one second tag item), which describes the anchor in such a way that the anchor can be easily identified by the user as a potential identifier and can be easily recognized and interpreted by the mobile augmented reality device.

[0039] Anchor identifiers are relatively unique, meaning they appear only once within a bubble (or at least once in a prominent position, or only a few times). Such anchor identifiers can be, for example, the text of an identifier label in a physical plant. These are commonly found in chemical, pharmaceutical, or power plants.

[0040] In addition, anchor identifiers can be, for example, text on street signs, posters or other large signs, the content of a label's barcode or QR code, the category of an object returned by an image processing algorithm (e.g., "cactus" could be an identifier if there is only one cactus in a bubble) and / or the category of an object returned by a 3D object detection algorithm (e.g., "pump type XYZ" could be an identifier if there is an algorithm that can detect and classify all types of pumps in a processing plant).

[0041] Information elements can also be referred to as holograms. In the following text, a hologram is understood as a container used to precisely place and enhance technical information relative to a real device, such as a component in physical infrastructure. Users can place, edit, modify, delete, and / or retrieve / view holograms.

[0042] Holograms can be user-created annotations. These annotations can include speech (audio and speech-to-text); floating 3D models, such as arrows; drawings; photos and videos captured from the device; or other documents. Holograms can be, for example, text, 3D models, small animations, instruction documents, photos, videos, etc.

[0043] Holograms can also include links to real-time data, such as graphs from temperature sensors inside the machine, or historical data.

[0044] Each hologram has a 3D position in a bubble coordinate system, such as the coordinate system of at least one second spatial environment.

[0045] Simultaneous Localization and Mapping (SLAM) is a computational problem that involves building or updating a map of an unknown environment while simultaneously keeping track of the location of an agent within it.

[0046] According to another embodiment, for example, at least one information element of the hologram is configured to be modified, added, and / or deleted by a user and / or machine. The advantage of this embodiment is that the content of the information elements used to enhance the bubble environment / secondary space environment can be adapted to the user's needs and preferences, and to changing environmental settings in the physical infrastructure or at least one secondary space environment.

[0047] According to another embodiment, enhancing the physical infrastructure includes using annotations, text, 3D models, charts, animations, moving 3D models, instruction documents, photographs, videos, arrows, drawings, live data, and / or links to live data. The advantage of this embodiment is that it provides users with an intuitive representation of information elements.

[0048] According to another embodiment, a method is provided that includes the step of providing enhanced audio output and / or voice output to at least one information element. Providing audio output and / or voice output has the advantage of helping users perform actions at accurate locations and increasing the accuracy of these actions.

[0049] According to another embodiment, the at least one first marker identifier and / or the at least one second identifier includes text, numeric codes, text of identifier labels, text of street signs, posters, signs, barcodes, QR codes, object categories such as those returned by image processing algorithms, and / or object categories such as those returned by 3D object detection algorithms. An advantage of this embodiment is the existence of different options for selecting at least one first marker identifier and / or at least one second identifier.

[0050] According to another embodiment, the at least one second spatial environment can be located by GPS, WiFi, and / or a positioning mechanism. The advantage of this embodiment is that the user equipment can determine, for example, its GPS location, and provide feedback to the user when, for example, the GPS location is accurate enough to, for example, create a bubble.

[0051] According to another embodiment, the at least one second spatial environment includes at least one spatial environment identifier unique within the first spatial environment. The spatial environment identifier may also be referred to as a bubble identifier. The bubble identifier allows for easier identification of at least one second spatial environment / bubble among a set of other second spatial environments / bubbles.

[0052] According to another embodiment, the at least one spatial environment identifier / bubble identifier includes a room name label, object, serial number, RF beacon, NFC tag, and / or WiFi access point fingerprint. The advantage of this embodiment is that at least one second spatial environment / bubble can be more easily identified from a set of other second spatial environments / bubbles.

[0053] According to another embodiment, at least one spatial environment identifier / bubble identifier is equal to at least one first marker identifier and / or at least one second identifier / anchor identifier.

[0054] Spatial context identifiers / bubble identifiers can be the same type of information used for anchor identifiers / first marker identifiers and second marker identifiers, such as barcodes, image classification results, etc., but the set of selected bubble identifiers / spatial context identifiers can differ from the set of anchor identifiers. This is because the set of bubble identifiers / spatial context identifiers should be relatively unique across the entire set of bubbles within a large area (e.g., a factory or a 200m x 200m outdoor area). For example, in an office building, "kitchen" could be such a bubble identifier. Furthermore, bubble identifiers / spatial context identifiers do not need to be relatively unique within a bubble, as they are to anchor identifiers / first marker identifiers and second marker identifiers. Therefore, "printer" could be a good bubble identifier / spatial context identifier for a printer room in an office building containing many printers, but not a good anchor identifier / first marker identifier and second marker identifier.

[0055] According to another embodiment, the at least one Simultaneous Localization and Mapping (SLAM) is adapted to varying settings within at least one second spatial environment. This means that if the characteristics of the physical environment change (e.g., light, objects, etc.), the SLAM also adapts to these changes. An advantage of this embodiment is that the physical infrastructure and the local optical characteristics of the at least one second spatial environment are known.

[0056] According to another embodiment, an augmented reality engine is provided for enhancing physical infrastructure using at least one information element. The augmented reality engine includes at least one processor and at least one data storage device.

[0057] The data storage device has computer-executable program code stored thereon, which, when executed by the processor, enables the augmented reality engine to: - Establish a first spatial environment within the physical infrastructure, the dimensions of which are determined by the coordinate system of the first spatial environment. - At least one second spatial environment is defined as a partial subspace of the first spatial environment, the size of the at least one second spatial environment is determined by the coordinate system of the at least one second spatial environment, and the at least one second spatial environment includes at least one first label item and at least one second label item. - Load or create at least one simultaneous localization and mapping (SMR) map, the size of which is determined by the coordinate system of the at least one SMR map. - Align the coordinate system of the at least one simultaneous positioning and mapping map with the coordinate system of the at least one second spatial environment. - Identify at least one first tag identifier and at least one second tag identifier, wherein the at least one first tag identifier corresponds to the at least one first tag item, and the at least one second tag identifier corresponds to the at least one second tag item. - By aligning the coordinate system of the first spatial environment with the coordinate system of the at least one second spatial environment using the at least one first marker and the at least one second marker, the absolute position and orientation of the at least one second spatial environment relative to the first spatial environment can be determined. - Define the at least one information element associated with at least one component of the physical infrastructure. - Determine the spatial coordinates of the information element in the coordinate system of the at least one second spatial environment. - Enhance the physical infrastructure at the determined spatial coordinates using at least one information element.

[0058] According to another embodiment, a mobile device including the augmented reality engine is provided.

[0059] This mobile device supports local SLAM algorithms and image capture, a GPS module, a wireless network module, and rendering and display capabilities. These devices are utilized by AR applications / methods that enable the creation of bubble / secondary spatial environments and the placement of annotation / information elements, as well as the retrieval and alignment of bubble / secondary spatial environments and the display of annotation / information elements.

[0060] According to another embodiment, the mobile device (client device or user device) is a smartphone, tablet computer, head-mounted display device, augmented reality head-mounted device, and / or virtual reality head-mounted device.

[0061] According to another embodiment, the mobile device has wide area network connectivity (through itself, such as a smartphone, or via another device, such as a holographic lens communicating with a smartphone as a mobile hotspot).

[0062] According to another embodiment, the mobile device includes a camera, a screen, and appropriate sensory hardware to enable augmented reality interaction.

[0063] According to another embodiment, the mobile device includes software capable of performing local SLAM (simultaneous localization and mapping) for augmented reality. Specifically, the SLAM software can create a 3D SLAM map of the local optical features of the world around it and save this map to a server. Furthermore, it can retrieve a pre-stored map of features from the server and then use that map for tracking. This means the augmented reality device "knows" its own position and orientation within a (arbitrary) coordinate system of the SLAM map. The size of this map is limited to a certain 3D area, for example, approximately 10 × 10 × 10 meters.

[0064] According to another embodiment, the mobile device can also perform image recognition tasks on the environment using photographs of it or send them to a server for image recognition tasks. It uses this to detect and classify identifiers (e.g., anchor identifiers / first marker identifiers and second marker identifiers, and bubble identifiers / spatial environment identifiers). This detection and classification of identifiers can also utilize 3D information, also used by SLAM software, for example, for 3D object classification.

[0065] Furthermore, SLAM software on mobile devices can estimate the distance of identifiers (e.g., anchor identifiers / first marker identifiers and second marker identifiers, and bubble identifiers / spatial environment identifiers) from the camera, thereby calculating the position of each identifier in the coordinate system of the SLAM software, the coordinate system of the first spatial environment, and the coordinate system of at least one second spatial environment.

[0066] According to another embodiment, the mobile device also includes GPS, WiFi-based, or similar geolocation devices. This allows it to determine its location with a certain precision, such as approximately 5 meters outdoors or approximately 50 meters indoors. (If the mobile device itself does not include a GPS sensor, it can be attached to another device, such as a holographic lens that communicates with a smartphone acting as a mobile hotspot and that the smartphone has GPS.)

[0067] According to another embodiment, a mobile device can display annotation / information elements in augmented reality within at least one second spatial environment retrieved from a server, and allow a user to view (or listen to) said annotation / information elements. These annotation / information elements may include speech (audio and speech-to-text); floating 3D models, such as arrows; drawings; photos and videos captured from the device, or other documents.

[0068] According to another embodiment, for information elements serving as live annotations, which display data, for example, from a SCADA system, mobile devices can connect to a local controller or edge device via a local network, or connect to a live IoT data aggregation platform via the cloud. The same applies to visualizing historical data, such as trends.

[0069] According to another embodiment, a server is provided that manages AR bubble / second space environment and its associated metadata, such as SLAM maps, and can provide image processing capabilities, such as image recognition, OCR, or 3D object detection. The server can also provide a web interface to place additional holograms / information elements into the bubble / second space environment.

[0070] The server can be a cloud server, a local server, or an edge server. The server stores data for its clients in the form of an "augmented reality bubble" / bubble / second space environment.

[0071] According to another embodiment, each bubble / second space environment represents data for at least one second space environment with a diameter of approximately 10 m. Furthermore, each bubble / second space environment has a geographical location defined by its GPS coordinates (or by its coordinates in another large-scale coordinate system, such as a factory coordinate system or building coordinate system / the coordinate system of at least one second space environment).

[0072] According to another embodiment, each bubble / second space environment has at least one coordinate system / coordinate system of the second space environment, which has an (arbitrarily defined) origin, preferably within the bubble / second space environment itself. The axes of the bubble coordinate system / coordinate system of at least one second space environment can be arbitrary, but are usefully defined such that one of the axes points upwards, which can be easily detected by an accelerometer.

[0073] According to another embodiment, each bubble / second space environment has a name entered by the user who created it, such as "Computer Room 33", as well as some representative photos that will make it easy for other users to identify the bubble / second space environment.

[0074] According to another embodiment, each bubble / second spatial environment has a list of anchor points / first markers and second markers, specifically having the type and data (e.g., "text ABCD", "barcode 12345", "image category cactus") for each anchor point identifier / first marker identifier and second marker identifier, as well as the 3D position of each identifier within the coordinate system of the bubble / coordinate system of at least one second spatial environment. For each anchor point / first marker and second marker, the server may also store one or more photographs representing the objects of the anchor point / first marker and second marker.

[0075] According to another embodiment, each bubble / second spatial environment has one or more stored spatial SLAM maps generated by the client device / mobile device to allow for accurate tracking. Each SLAM map consists of a set of binary data generated by a SLAM algorithm on the mobile device and subsequently reinterpreted. Furthermore, for each SLAM map of the bubble / second spatial environment, the server stores the relative position and orientation of the origin of that SLAM map with respect to the actual coordinate system of at least one bubble / coordinate system of the second spatial environment (in cases where the SLAM map origin is not equal to the bubble origin).

[0076] Furthermore, within each SLAM map in the bubble / second space environment, the server stores metadata to help determine which of a given set of SLAM maps is likely optimal for a given augmented reality device at a given time. This metadata may include, for example, the time and date the SLAM map was created; the type of device or the version of the SLAM software used to create it; or the type of lighting active in the bubble at the time, such as sunlight or artificial lighting, or even weather data, in the case of external applications.

[0077] According to another embodiment, each bubble / second spatial environment optionally includes a reference to a bubble identifier / spatial environment identifier, making it easier for the client / user to identify it among a set of other bubble / spatial environments. This can be the same type of information as that used for anchor identifiers / first marker identifiers and second marker identifiers (barcodes, image classification results, etc.), but the selected set of bubble identifiers can be different from the set of anchor identifiers / first marker identifiers and second marker identifiers. This is because the set of bubble identifiers should be relatively unique over the entire bubble set within a large area (e.g., a factory or a 200m × 200m outdoor area). For example, in an office building, "kitchen" could be such a bubble identifier / spatial environment identifier. Furthermore, the bubble identifier / spatial environment identifier does not need to be relatively unique within the bubble / spatial environment like the anchor identifier / first marker identifier and second marker identifier, such that "printer" could be a good bubble identifier / spatial environment identifier for a printer room in an office building containing many printers, but not a good anchor identifier / first marker identifier and second marker identifier.

[0078] According to another embodiment, each bubble / second space environment contains references to all data of the hologram within the bubble / second space environment—text, audio, photos, videos, documents, etc.

[0079] If the client / user does not perform these tasks entirely independently, the server also provides the necessary image recognition and / or 3D object detection capabilities to process the data uploaded by the client.

[0080] The server has a search engine that allows clients to query the search engine to determine which AR bubble / second space environment they are in.

[0081] Servers typically also have user management and security features to allow access control. Different users may be allowed to view and / or edit only certain bubble / second space environments, or certain hologram / information elements within certain bubble / second space environments, or certain types or groups of hologram / information elements.

[0082] Optionally, the included embodiments have non-AR client devices, such as PCs, that allow users to perform administrative tasks, such as access control or bulk data import. Specifically, this can include data import tools to import data from existing 3D models, such as BIM systems.

[0083] Optionally, the included embodiments include software and printing equipment that, if there are not enough existing labels or other identifiers (e.g., anchor identifiers / first mark identifiers and second mark identifiers, as well as bubble identifiers / space environment identifiers) within the bubble / second space environment, allows a user to generate and print relatively unique labels attached to areas of the bubble / second space environment.

[0084] According to another embodiment, a computer program product is provided. The computer program product includes instructions that, when executed by a computing device, cause the computing device to perform the method according to the embodiment.

[0085] According to another embodiment, a computer-readable storage medium is provided. The computer-readable storage medium includes instructions that, when executed by a computing device, cause the computing device to perform the method according to the embodiment.

[0086] Other systems have attempted to address the problem of persistently registering augmented reality (AR) information by saving simple SLAM maps or by using laser scanners and cameras to scan areas and then performing server-side image processing to align the camera images of the augmented reality device with the point cloud. All existing methods require sophisticated server-side image processing software, specialized scanning equipment, or are inaccurate, slow, or fragile in response to changes in lighting.

[0087] The embodiments disclosed herein use the following unique combination: - Subdivide the world / physical infrastructure space into bubbles / sub-spaces of appropriate size for the SLAM capabilities of current AR devices. - Anchor points / first and second marker identifiers based on identifiers, which are easily recognizable to humans and AR devices, are often already available as real-world signs or labels, and frequently correspond to existing identifiers within the 3D model. - Utilize the SLAM capabilities of today's devices to quickly map small areas using the AR device itself, without needing expensive infrastructure such as laser scanners, and - The incremental mapping process enables frequently accessed areas to quickly view AR data without incurring overhead for unaccessed areas; it also makes it easy to adapt to changes in lighting and environment over time.

[0088] The embodiments disclosed herein have several advantages that make them cost-effective and robust, especially in industrial environments.

[0089] The first advantage is interoperability: systems using a closed approach can utilize SLAM algorithms for different device types and from different vendors.

[0090] Another advantage is low server load. Because the implementation fully utilizes current device capabilities (SLAM, image processing), the server does not have to perform computationally intensive tasks; it simply provides shared persistence and image processing for those devices that need it.

[0091] Another advantage is that the implementation can operate in offline mode (without a server), downloading site bubble, anchor point, SLAM map, and hologram information to a mobile AR device before visiting the site, and performing (simple) image recognition tasks on the AR device. Subsequently, changes to the hologram, anchor points, and SLAM map are uploaded; these changes are made on-site.

[0092] Another advantage is that the embodiments can operate with or without GPS. If GPS is not available, the invention can use a set of globally unique identifiers to identify bubbles, or use contextual data from another source (e.g., the name of the factory where the user is working).

[0093] Another advantage is that the server does not need to understand the data it is persistently storing; therefore, SLAM maps, bubble metadata, and hologram data can be encrypted, for example, using a shared symmetric key that can only be accessed by customer devices on a specific factory site.

[0094] Another advantage is that no special scanning or mapping equipment is required. The implementation can operate entirely on current smartphones and servers; no laser scanners, camera carts, drones, or similar equipment are needed.

[0095] Another advantage is the cost-effective incremental shared mapping. Many bubbles can be automatically generated from an existing 3D factory model; and after a user's initial visit, each bubble will have a SLAM map that other users can share.

[0096] Another advantage is the very low initial deployment cost. Since no special hardware is required and regions can be mapped incrementally, the initial cost of deploying this invention in a given region is very low.

[0097] Another advantage is the low cost of updates. Updates to bubble and anchor data can be performed on-site by users as they use the system, thus reducing costs.

[0098] Another advantage is that the implementation allows for the combination of data from existing models and data from the field. Field users can add holograms and anchor points to bubbles created off-site from existing 3D models using the import tool. They can also create their own new bubbles, for example, within sections of a factory site that do not currently have a 3D model. Subsequent users will then be able to access data from both types of bubbles.

[0099] Another advantage is that the implementation allows for the rapid display of AR content in most cases. When a user loads data for a bubble, in most cases, the working SLAM map of that bubble will have been created and stored by another user before them, allowing the AR client device to quickly locate itself.

[0100] Another advantage is that the embodiments allow for robust location identification. Because the bubble identification is based on scanning "relatively unique" identifiers, the identified bubble is likely to be correct.

[0101] Another advantage is that the implementation supports robustness against offset errors in repetitive environments. Some environments can appear very repetitive, such as a corridor filled with identical meeting rooms. Simply relying on image recognition or SLAM maps could result in AR content being displayed in the wrong room. By scanning the identifiers of the bubbles (and anchor points), this is much less likely to happen.

[0102] In one variant, 3D data generated by manually creating bubbles on-site can be fed into a BIM or asset management system to create a new geospatial model of the object corresponding to the anchor point using their identifiers and images.

[0103] In another variation, the detection of bubble identifiers and anchor point identifiers, as well as the loading of GPS-based bubbles, can all be performed with minimal prompts and without user intervention. For example, an AR device could prompt the user to “slowly move your phone around and point it at different objects, especially text labels,” and after about 20 seconds, the process of identifying and inputting bubbles will be completed, regardless of whether a SLAM map is available.

[0104] In another variation, the embodiment can be extended by using a technique that allows users to move seamlessly from one bubble to another, which is achieved by aligning their respective coordinate systems based on the GPS location of the bubbles and / or anchor points shared between adjacent bubbles and / or a general 3D model including the anchor points.

[0105] In another variation, the implementation can be extended to work with bubbles in non-geographic reference coordinate systems or moving bubbles. For example, instead of a GPS coordinate system, a system can be established for a factory site that operates entirely within a "factory" coordinate system. Furthermore, the factory will be subdivided into bubbles, and the bubble identification will be based on bubble identifiers. Alternatively, the system can be set up to operate in moving or mobile environments, such as trains, ships, or airplanes, with bubbles representing different parts of the vehicle. Once the system has identified which vehicle the AR device is in (by manually entering or scanning an identifier), the system operates entirely within a vehicle coordinate system, which is also subdivided into bubbles.

[0106] In another variant, the system can also use RF beacons, NFC tags, or WiFi access point fingerprint recognition to detect bubble identifiers. To detect anchor point identifiers, the system can also use NFC tags—because these tags operate at close range, the camera's SLAM position estimation will be sufficient to estimate the location of the corresponding anchor point.

[0107] The following provides an exemplary interaction between the user and the appended embodiments.

[0108] According to another embodiment, the user interacts with the method according to the embodiment as follows. To manually create a new bubble on-site, the user wants to create new augmented reality content and place it where there was no such content previously. To do this, they activate the "Create Bubble" function on the client device.

[0109] The client device determines its GPS location and provides feedback to the user when the GPS location is accurate enough to create a bubble (alternatively, another positioning mechanism, such as WiFi, can be used instead of GPS).

[0110] The user enters the name of the bubble (and optionally other relevant metadata). The client device prompts the user to take photos of visually interesting elements of the environment (such as room name labels, serial numbers, posters, etc., as mentioned above) to create bubble identifiers to help resolve ambiguity later, and to present some overview photos of the bubble to other users.

[0111] As users walk through the bubble's area, the AR client device has ample opportunity to create a detailed SLAM map of that area. Furthermore, users can point their camera at anchor markers such as text labels, barcodes, or relatively unique object identifiers.

[0112] The client performs image processing itself to detect and label these bubble identifiers and anchor point identifiers, or sends the image to the server for processing. It can also use the feature point distance estimation function of the SLAM algorithm to calculate the 3D position of the anchor points relative to the (arbitrary) origin of the SLAM map. Optionally, it also records a photograph of each anchor point to help the user manually label these anchor points later.

[0113] Once the client device has created a sufficiently detailed map and recorded enough anchor points (at least two), the device informs the user and uploads the SLAM map, anchor point data, and all other bubble data to the server.

[0114] In this case, the origin of the bubble coordinate system is the origin of the coordinate system of the first SLAM map.

[0115] According to another embodiment, the user interacts with the method according to the embodiment as follows. To create a bubble from a pre-existing digital model, a pre-existing digital model of the device within the bubble exists, such as from a BIM system, a factory planning system, a 3D processed factory model, or even a 2D map. The user selects an appropriate amount of data from this 3D model and uploads the data to the server in the form of a new bubble using an import tool running on a non-AR client device, such as a PC.

[0116] The import tool creates a new bubble on the server, sets an appropriate friendly name, and generates a preview image that can be presented to users on-site later.

[0117] The import tool also creates a list of bubble identifiers based on the equipment list in the existing 3D model. For example, if the BIM system identifies a room as "Meeting Room 3", and users know that, by convention, all meeting rooms in a building have room labels on their doors, they would add the identifier "Text Label Room 3" to the list of bubble identifiers.

[0118] The import tool creates an (arbitrary) origin for the bubble coordinate system; this will typically be inside the bubble and can follow conventions such as the center of a meeting room or the nearest point on a 10m × 10m grid. The import tool sets the bubble's GPS position to be equal to the origin of the bubble coordinate system (assuming that the GPS position is known). The GPS position can be approximate, for example, accurate to only 100m, depending on the availability of good map data.

[0119] The import tool adds a set of anchor points with 3D locations and anchor point identifiers to the bubble, based on a list of devices in the existing 3D model. For example, if the BIM system identifies an Ethernet port in a room as "Port 3 / 57," and users know that by convention all Ethernet ports in a building are labeled, they would add the identifier "Text Label Port 3 / 57" to the anchor point list, along with the 3D location of that Ethernet port. The import tool then uploads all this bubble data to the server.

[0120] The import tool then (optionally) generates a set of holograms as AR information to be displayed to the user within the bubble, and also uploads them to the server, where the position and orientation are defined in the bubble's coordinate system. For example, the holograms could include maintenance information for a printer in a printing room and could be positioned above the printer's location defined in the BIM model.

[0121] According to another embodiment, the user interacts with the method according to the embodiment as follows. To identify the bubble, the user activates the "Find Bubble" function on the client device.

[0122] The client device determines the user's approximate geographic location and queries the server for nearby location bubbles. This can include a radius, such as 200m, because GPS location queries, for example, may be inaccurate indoors.

[0123] If more than one possible bubble exists within the accuracy range of the GPS location, the device prompts the user to point the camera at something that will be used as a bubble identifier. This could be, for example, a printer in a print room as described above, or a text label or marker with a room number. It could also be an anchor point identifier, but it is not required.

[0124] The client then performs image processing itself to detect and identify these identifiers, or sends the image to the server for processing. Using the resulting set of bubble identifiers, the client then performs a query to the server to refine the GPS-based query.

[0125] If there is still more than one possible bubble, the process continues, during which the user points the camera at more objects.

[0126] The client software can also present users with a (shorter) list of possible bubbles, along with friendly names and logo images, allowing users to manually select the bubble they are in.

[0127] At the end of the process, the augmented reality device can know exactly which bubble it is in.

[0128] According to another embodiment, the user interacts with the method according to the embodiment as follows. In order to input bubbles using SLAM maps, once the client device knows which bubble the user is in, it loads the corresponding set of anchor points and at least one appropriate SLAM map from the server (appropriate means that the SLAM map was created by the same SLAM software on a similar device, not too long ago, and under similar lighting conditions).

[0129] If no SLAM map is available, for example, because there are no users with this type of augmented reality device on site, see “Entering the bubble without a SLAM map” below.

[0130] The SLAM algorithm then attempts to reposition the augmented reality device based on the SLAM map. To do this, it prompts the user to move a little, giving the device the opportunity to map out the environment and match it to the stored SLAM map.

[0131] If the SLAM relocalization algorithm succeeds, it returns the camera's position and orientation relative to the origin of the SLAM map. Using this information, along with the position and orientation of the stored SLAM map's origin relative to the bubble's origin, the augmented reality device calculates its position and orientation relative to the bubble coordinate system, and can now display the hologram in the correct position and orientation.

[0132] Optionally, the AR client device can also refine the SLAM map based on the current lighting conditions and save it as a new additional SLAM map to the server.

[0133] According to another embodiment, the user interacts with the method according to the embodiment as follows. When entering a bubble without a SLAM map, if SLAM repositioning is unsuccessful (e.g., because furniture in the room has been moved or lighting has changed), or if there is no stored SLAM map for that device type, the SLAM algorithm will be able to identify the device's position and orientation relative to any new coordinate system it has created, but it will not be able to match it to a stored SLAM map.

[0134] In this scenario, the augmented reality device prompts the user to point the camera at an anchor identifier, such as a text label, barcode, or a relatively unique object.

[0135] The client then performs image processing itself to detect and identify these identifiers, or sends the image to the server for processing. It also uses feature point distance estimation from the SLAM algorithm to determine the location of these identifiers (within the coordinate system of the new SLAM map) in 3D space.

[0136] Alternatively, AR client devices can present users with a list of photos of anchor points and ask them to point the AR device at the location of the corresponding object, thus positioning them in 3D space. This is useful when anchor objects are difficult for algorithms to identify but easy for humans to identify.

[0137] Once enough anchor points have been identified (at least two), the client device performs geometric operations to calculate a rotation and translation that optimally aligns the measured 3D position of the identified anchor points with the stored 3D position of the same anchor points from the server. (Assuming the client device has an accelerometer capable of accurately measuring the direction of gravity, this transformation can be restricted to, for example, vertical rotation.)

[0138] This transformation now determines the position and orientation of the origin of the new SLAM map relative to the bubble's origin. As a result, the augmented reality device calculates its position and orientation relative to the bubble's coordinate system and can now display the hologram in the correct position and orientation.

[0139] Optionally, the device can then store the newly created SLAM map along with its newly calculated transformations and metadata on a server so that this can be used for rapid relocation in the future.

[0140] According to another embodiment, the user interacts with the method according to the embodiment as follows. To refine the bubble by updating / adding anchor points, when the user has entered the bubble, they can choose to update the location, identifier, or photos of the anchor points within the bubble. This is particularly useful if the bubble has already been created from a pre-existing digital model.

[0141] For example, one of the anchor points in the bubble might be incorrect because it was generated from a planned BIM model, but the construction of the equipment in the room differs slightly. In this case, the AR client device can visualize the location and identifiers of all the stored anchor points within the bubble, and the user can interactively move the individual anchor points around to align them with real objects (or their text labels).

[0142] In addition, users can delete anchor points for objects that simply don't exist. Furthermore, users can create new anchor points to add to the bubble. For example, the import tool might have already generated anchor points for all Ethernet ports in a room, and a site user might choose to add anchor points for a sign that is not in the BIM model but is easily identifiable.

[0143] Specifically, users might want to attach new labels with arbitrarily generated text labels or QR codes to a wall or a device. To do this, they could print a unique set of labels using label printing software before visiting the venue, then randomly select one and stick it to the wall. They would then use an AR client device to scan the new label and add it to a list of anchor points within a bubble.

[0144] Adding new anchors is particularly useful if all existing anchors correspond to small, hard-to-read tags, making it even more difficult to enter bubbles without a SLAM map. Users can add new, easy-to-read tags, making it easier to enter bubbles without a SLAM map in the future.

[0145] According to another embodiment, the user interacts with the method according to the embodiment as follows. In order to bring content from an existing 3D site model into the site (e.g., from a model of a processing plant or a BIM model), multiple bubbles (e.g., for each room or every 10 meters) are first created for the plant / building using the process described in "Creating Bubbles from a Pre-existing Digital Model," where appropriate holograms for the data are to be displayed.

[0146] Second, send users (or multiple users) to the actual factory / building site. Select all areas of the site that the users will frequently visit. The user walks around the site using a type of AR client device (e.g., a mobile phone), and then for each bubble in that frequently visited area, identify the bubble, enter the bubble without a SLAM map (because there is no SLAM map yet), and store the SLAM map for that type of device and optionally refine the bubble by adding additional anchor points.

[0147] From then on, any user with this type of AR client device will be able to quickly and easily enter bubbles with SLAM maps.

[0148] Users accessing "infrequently visited areas" will not be able to enter bubbles with SLAM maps first. Instead, for each bubble in such an infrequently visited area, the user will have to enter a bubble without a SLAM map once; subsequent users will then be able to enter bubbles with SLAM maps.

[0149] According to another embodiment, the user interacts with the method according to the embodiment as follows. To introduce a novel AR client device to the site, for example, once AR content has been generated and placed using a smartphone, a holographic lens is introduced into the processing plant, sending one or more users to frequently accessed areas of the site. For each bubble, the bubble can be identified using the new AR client device, entering bubbles without a SLAM map, and a newly created SLAM map for that device is saved.

[0150] Again, as a result, in the frequently visited area, users will be able to use their new type of AR device with SLAM maps, while in all other areas, the first user of a bubble will have to enter a bubble without a SLAM map first.

[0151] According to another embodiment, the user interacts with the method according to the embodiment as follows. To update the bubble to reflect changes in the appearance, lighting, or position of objects with anchor points, any user entering a bubble with or without a SLAM map can save the new SLAM map to the server. (This can even be done automatically within the AR client device).

[0152] Any user with the appropriate permissions can choose to refine the bubble to update the position of existing anchors (this can even be automatic within the AR client device, for example, if it detects that an anchor has moved significantly within the bubble relative to other anchors).

[0153] According to another embodiment, the user interacts with the method according to the embodiment as follows. In order to combine data from an existing model and data input from the site, a bubble for the building in the site with the digital model is first created using the process described in "Creating a Bubble from a Pre-existing Digital Model".

[0154] Then, bubbles are manually created on-site in areas of the site where there are no building models yet (such as outdoor areas). Users will be able to persistently place and view AR content in both types of areas. Attached Figure Description

[0155] Figure 1 This shows a portion of the physical infrastructure aligned with marker items and enhanced with information elements. Detailed Implementation

[0156] Figure 1 A portion of the physical infrastructure subspace is shown, such as the second spatial environment 2, or the first spatial environment 1, which is aligned by the first marker item 5 / the second marker item 5 and enhanced by the information element 4.

[0157] The first spatial environment 1 is established within the physical infrastructure 1. The dimensions of the first spatial environment 1 are determined by the coordinate system of the first spatial environment.

[0158] The second spatial environment 2 is defined as a partial subspace of the first spatial environment 1. The size of the second spatial environment 2 is determined by the coordinate system of the second spatial environment 3.

[0159] Load at least one simultaneous localization and mapping map. The at least one simultaneous localization and mapping map includes eleven markers 5 (for each second spatial environment 2, at least one first marker 5 and at least one second marker 5 are required).

[0160] Each second spatial environment 2 is identified by at least one first tag identifier and at least one second tag identifier. The at least one first tag identifier corresponds to at least one first tag item 5, and the at least one second tag identifier corresponds to at least one second tag item 5.

[0161] The absolute position of the second spatial environment 2 relative to the first spatial environment 1 is determined by aligning the coordinate system of the first spatial environment 2 with the coordinate system of the second spatial environment 3 using at least one first marker 5 and at least one second marker 5.

[0162] At least one information element 4 is defined in relation to at least one component of the physical infrastructure.

[0163] Define the spatial coordinates of information element 4 in the coordinate system of the second spatial environment 3.

[0164] The physical infrastructure is enhanced at the determined spatial coordinates using five information elements 4.

[0165] Although the invention has been explained with reference to the advantageous embodiments described above, it should be understood that many other possible modifications and variations can be made without departing from the scope of the invention. Therefore, it is contemplated that the appended claims should cover these modifications and variations that fall within the true scope of the invention.

[0166] List of reference numerals 1. First Space Environment 2 Second Space Environment 3. At least one coordinate system of a second spatial environment 4 Information Elements 5. First marker item / Second marker item.

Claims

1. A computer-implemented method for enhancing physical infrastructure using at least one information element, the method comprising the following computer-implemented steps: Obtain 3D data of a first spatial environment within the physical infrastructure, wherein the size of the first spatial environment is determined by a coordinate system of the first spatial environment. At least one second spatial environment is defined as a partial subspace of the first spatial environment, the size of the at least one second spatial environment is determined by the coordinate system of the at least one second spatial environment, and the at least one second spatial environment includes at least one first label item and at least one second label item. Create at least one simultaneous localization and mapping (SLAM) map of the at least one second spatial environment, the at least one SLAM map being sized by the coordinate system of the at least one SLAM map, wherein the at least one SLAM map contains at least two SLAM tag items, each having a corresponding SLAM tag identifier within the coordinate system of the at least one SLAM map; Identify SLAM tag identifiers; Align the coordinate system of the at least one SLAM map with the coordinate system of at least one second spatial environment; Identify at least one first tag identifier and at least one second tag identifier, wherein the at least one first tag identifier corresponds to the at least one first tag item and the at least one second tag identifier corresponds to the at least one second tag item. By matching the at least one first marker and the at least one second marker to align the coordinate system of the first spatial environment and the coordinate system of the at least one second spatial environment, the absolute position and orientation of the at least one second spatial environment relative to the first spatial environment are determined. Define the at least one information element associated with at least one component of the physical infrastructure. Determine the spatial coordinates of the information element in the coordinate system of at least one second spatial environment. The physical infrastructure is enhanced using at least one information element at the determined spatial coordinates.

2. The method according to claim 1, in, The at least one information element is configured to be modified, added, and / or deleted by a user and / or a machine.

3. The method according to any one of claims 1-2, in, Enhancing the physical infrastructure includes strengthening it through the following: Notes text, 3D model, chart, Animation, Moving 3D models Instruction file, photo, video, arrow, Drawing, Real-time data, and / or Link to live data.

4. The method according to any one of claims 1-2, This includes the step of providing enhanced audio and / or voice output for the at least one information element.

5. The method according to any one of claims 1-2, in, The at least one first tag identifier and / or the at least one second tag identifier include: text, Numerical code, The text of the identifier label, The text of the street sign, Posters logo, Barcode, QR code The category of the object returned by the image processing algorithm, and / or The category of the object returned by the 3D object detection algorithm.

6. The method according to any one of claims 1-2, The at least one second spatial environment can be located via GPS, WiFi, and / or a positioning device.

7. The method according to any one of claims 1-2, The at least one second spatial environment includes at least one spatial environment identifier that is unique within the first spatial environment.

8. The method according to claim 7, in, The at least one spatial environment identifier includes: Room name label object, Serial number, RF beacon, NFC tags, and / or Fingerprint access point for WiFi.

9. The method according to claim 7, in, The at least one spatial environment identifier is equal to the at least one first tag identifier and / or the at least one second tag identifier.

10. The method according to any one of claims 1-2, in, The characteristics of the at least one Simultaneous Localization and Mapping (SLAM) map are suitable for the at least one second spatial environment.

11. An apparatus for enhancing physical infrastructure using at least one information element, the apparatus comprising: At least one processor; and; At least one data storage device having stored computer-executable program code thereon, the computer-executable program code causing the device to: A first spatial environment is established within the physical infrastructure, the dimensions of which are determined by the coordinate system of the first spatial environment. At least one second spatial environment is defined as a partial subspace of the first spatial environment, the size of the at least one second spatial environment is determined by the coordinate system of the at least one second spatial environment, and the at least one second spatial environment includes at least one first label item and at least one second label item. Create at least one simultaneous localization and mapping (SLAM) map, the size of which is determined by the coordinate system of the at least one SLAM map, wherein the at least one SLAM map contains at least two SLAM tag items, each having a corresponding SLAM tag identifier within the coordinate system of the at least one SLAM map. Identify SLAM tag identifiers; Align the coordinate system of the at least one SLAM map with the coordinate system of the at least one second spatial environment. Identify at least one first tag identifier and at least one second tag identifier, wherein the at least one first tag identifier corresponds to the at least one first tag item and the at least one second tag identifier corresponds to the at least one second tag item. By using at least one first marker and at least one second marker to align the coordinate system of the first spatial environment and the coordinate system of the at least one second spatial environment, the absolute position and orientation of the at least one second spatial environment relative to the first spatial environment are determined. Define the at least one information element associated with at least one component of the physical infrastructure. Determine the spatial coordinates of the information element in the coordinate system of at least one second spatial environment. The physical infrastructure is enhanced using at least one information element at the determined spatial coordinates.

12. A mobile device comprising the means according to claim 11.

13. The mobile device according to claim 12, comprising: Smart phone Tablet PC Head-mounted display devices, Augmented reality headsets, and / or Virtual reality headset.

Images