Method for generating a map for an augmented reality device in an industrial facility

Abstract

A method for generating a map for an augmented reality device in an industrial facility comprising two or more subspaces is described. Each of the two or more subspaces comprises a plurality of anchor points. The method comprises obtaining location information of a first subspace from the plurality of subspaces, determining a coordinate system of the first subspace based on the obtained location information and the plurality of anchor points of the first subspace; and calculating a relative position and orientation of at least one neighboring subspace based on the determined coordinate system for generating the map based on the location information of the first subspace and the relative position of the at least one neighboring subspace. The at least one neighboring subspace comprises one or more anchor points in an area overlapping with the first subspace.

Description

Background Technology

[0001] Augmented reality (AR) applications provide interactive experiences to users in real-world environments. Objects residing in the real-world environment are augmented with computer-generated information. The displayed overlay information can intertwine with the physical reality in augmented reality, allowing users to perceive it as an immersive aspect of the real environment. Augmented reality can be used to enhance natural environments or situations and provide users or operators with a sensorily rich experience. Augmentation techniques are typically performed in real time and within a semantic context that includes environmental elements or objects.

[0002] In many use cases, it is necessary to place augmented reality annotations (i.e., displayed overlapping information) relative to specific locations or objects in the physical real world. In industrial applications, this is particularly useful for information related to physical infrastructure. For example, regarding machine commissioning, servicing, and maintenance, relevant information such as the type of materials / parameters can be provided in advance and / or continuously annotated during commissioning, servicing, and maintenance activities.

[0003] There are many different methods for creating and displaying augmented reality (AR) content. One such method is tag-based AR display technology, where AR content is created in a 3D graphics programming environment and anchored to 2D visual tags. The AR content is then retrieved when the camera of a client / mobile device manipulated by the user reads the 2D visual tags. In another method, instead of 2D visual tags, real objects such as industrial equipment are scanned and detected by the client device, and AR content associated with the real objects is then retrieved and displayed. In yet another method, AR content is retrieved based on the location of the client device. For example, AR is georeferenced and acquired based on the location of the client device determined by location detection techniques such as GPS, wide-area RF positioning, etc. Another traditional method is tag-based optical tracking technology. Cameras on the AR device identify optical tags placed at different locations throughout the factory. Each optical tag is designed to be easily identifiable by image processing techniques. Software on the AR device detects the tags, thereby identifying which part of the factory the user is in; then pose estimation is performed based on the viewpoint from which the tag appears from the device's camera position. Thus, when a tag is detected, the AR device has a good estimate of its exact location within the factory. The drawback of this technology is that special markings must be placed in precisely defined locations within the factory.

[0004] Each of the techniques listed above has some drawbacks, such as in preparation, long-term durability of the markers, spatial accuracy, or interoperability between these techniques. Therefore, there is a need for a method and apparatus for precisely scaling up physical infrastructure at precise locations relative to actual devices. Summary of the Invention

[0005] Therefore, this invention describes a method for generating maps for augmented reality devices in an industrial facility comprising two or more subspaces. Each of the two or more subspaces includes a plurality of anchor points, wherein the plurality of anchor points of a respective subspace includes one or more anchor points located in a region overlapping with another subspace among the plurality of subspaces. The method includes acquiring positional information of a first subspace among the plurality of subspaces; determining a coordinate system of the first subspace based on the acquired positional information of the first subspace and the plurality of anchor points; and calculating the relative position and orientation of at least one adjacent subspace based on the determined coordinate system, for generating a map based on the positional information of the first subspace and the relative position of at least one adjacent subspace. The at least one adjacent subspace includes one or more anchor points in a region overlapping with the first subspace.

[0006] Therefore, this invention describes a method for generating maps that ensures the efficient generation of augmented reality displays. Errors caused by incorrect positioning are reduced by using the generated maps.

[0007] In one example, the method further includes obtaining location information of a first subspace among the plurality of subspaces. Therefore, the location information can be retrieved from a location server or via a positioning subsystem within a portable device. In one example, the method includes generating a graph associated with two or more subspaces, wherein the graph includes a plurality of nodes and edges, each of the plurality of nodes being associated with a subspace of the two or more subspaces and an anchor point of a first set of anchor points, wherein each anchor point of the first set of anchor points is located in at least one region overlapping with one of the two or more subspaces and another subspace.

[0008] In another aspect, the present invention describes a method for displaying data on an augmented reality device in an industrial facility comprising multiple subspaces. The method includes: determining a first subspace and the location and orientation of the augmented reality device within the first subspace, wherein the augmented reality device is located within the first subspace of the industrial facility; determining one or more other subspaces based on the orientation of the augmented reality device in the first subspace to display data associated with one or more objects in the one or more subspaces; wherein each of the one or more other subspaces includes at least one anchor point in a region overlapping one or more of the first subspace and another subspace of the one or more other subspaces; and filling the display of the augmented reality device with one or more data elements associated with one or more objects in the first subspace and the one or more other subspaces. In one example, the method further includes determining a first set of anchor points in the first subspace for determining the first subspace and the orientation and location of the augmented reality device within the first subspace, wherein each anchor point in the first set of anchor points is a physical identifier attached to a corresponding location within the first subspace.

[0009] In another example, the method also includes using one or more mapping sensors to generate a map for determining the orientation and location of the first subspace and the augmented reality device.

[0010] In another aspect, the present invention describes a server for generating maps for augmented reality devices in an industrial facility, the industrial facility comprising two or more subspaces, each of the two or more subspaces comprising a plurality of anchor points, wherein the plurality of anchor points of a respective subspace includes one or more anchor points located in a region overlapping with another subspace of the plurality of subspaces. The server includes one or more processors connected to a non-transient memory module. The one or more processors are configured to obtain location information of a first subspace of the plurality of subspaces; determine a coordinate system of the first subspace based on the obtained location information and the plurality of anchor points of the first subspace; and calculate the relative position and orientation of at least one adjacent subspace based on the determined coordinate system, for generating a map based on the location information of the first subspace and the relative position of at least one adjacent subspace, wherein the at least one adjacent subspace includes one or more anchor points in a region overlapping with the first subspace.

[0011] Similarly, the present invention also describes a non-transient storage medium for generating maps for augmented reality devices in an industrial facility, the industrial facility comprising two or more subspaces, each of the two or more subspaces comprising a plurality of anchor points, wherein the plurality of anchor points of a respective subspace includes one or more anchor points located in a region overlapping with another subspace of the plurality of subspaces. The non-transient storage medium includes a plurality of instructions, which, when executed on one or more processors, cause the one or more processors to acquire position information of a first subspace among the plurality of subspaces; determine a coordinate system of the first subspace based on the acquired position information and the plurality of anchor points of the first subspace; and calculate the relative position and orientation of at least one adjacent subspace based on the determined coordinate system, for generating a map based on the position information of the first subspace and the relative position of at least one adjacent subspace, wherein the at least one adjacent subspace includes one or more anchor points in a region overlapping with the first subspace.

[0012] Additionally, the present invention describes an augmented reality device for displaying data associated with one or more objects in an industrial facility comprising multiple subspaces. The augmented reality device includes one or more processors connected to a non-transient memory module, the processors being configured to determine a first subspace and the location and orientation of the augmented reality device within the first subspace, wherein the augmented reality device is located within the first subspace of the industrial facility; determine one or more other subspaces based on the orientation of the augmented reality device in the first subspace to display data associated with one or more objects in the one or more subspaces, wherein each of the one or more other subspaces includes at least one anchor point in a region overlapping one or more of the first subspace and another subspace of the one or more other subspaces; and populate the display of the augmented reality device with one or more data elements associated with one or more objects in the first subspace and the one or more other subspaces.

[0013] Additionally, the present invention describes a non-transient storage medium for displaying data associated with one or more objects in an industrial facility comprising multiple subspaces. The non-transient storage medium includes a plurality of instructions, which, when executed on one or more processors, cause the processors to determine a first subspace and the location and orientation of an augmented reality device within the first subspace, wherein the augmented reality device is located within the first subspace of the industrial facility; determine one or more other subspaces based on the orientation of the augmented reality device in the first subspace to display data associated with one or more objects in the one or more subspaces, wherein each of the one or more other subspaces includes at least one anchor point in one or more overlapping regions with the first subspace and another one or more other subspaces; and populate the display of the augmented reality device with one or more data elements associated with one or more objects in the first subspace and the one or more other subspaces. The advantages of the method apply to the device of the present invention. Figures 1-5 These aspects will be further described in the text. Attached Figure Description

[0014] The following detailed description refers to the accompanying drawings, in which:

[0015] Figure 1 An exemplary section of an industrial facility, including multiple industrial devices, is shown on an augmented reality device.

[0016] Figure 2 An example method for generating maps for augmented reality devices in an industrial facility is shown;

[0017] Figure 3 The logical representation of a portion of an industrial facility comprising three subspaces is shown;

[0018] Figure 4 This demonstrates a method for displaying data on augmented reality devices in industrial facilities; and

[0019] Figure 5 It shows relative to Figure 3 The example diagram shows the subspace generation. Detailed Implementation

[0020] Figure 1 Section 1, showing an industrial facility on an augmented reality device, is illustrated, comprising multiple industrial devices within industrial facility 100. In this invention, an industrial facility refers to any environment in which one or more industrial processes (e.g., manufacturing, refining, smelting, assembly of equipment) can be performed, and includes refineries, oil refineries, automobile factories, etc. The multiple industrial devices include industrial equipment, control equipment, field devices, mobile devices, operator stations, etc. Control equipment includes process controllers, programmable logic controllers, supervisory controllers, automated guided vehicles, robots, operator equipment, etc. One or more control devices are connected to multiple field devices. The multiple field devices include actuators and sensor devices for monitoring and controlling the industrial equipment in the industrial facility.

[0021] These field devices can include flow meters, valve actuators, temperature sensors, pressure sensors, etc. All industrial equipment can be interconnected via one or more networks (achieved through wired and wireless technologies).

[0022] Additionally, as mentioned above, industrial facility 100 may include augmented reality devices (…). Figure 1 (Not shown in the image) This augmented reality device (ARD) is used to display the status of one or more industrial devices in an industrial facility to an operator and allows the operator to define KPIs for controlling industrial processes within the facility. Based on the orientation and location of the ARD device, it can display multiple graphical elements associated with the industrial equipment within the ARD device's camera view. For example, as shown... Figure 1 As shown, when the augmented reality device's camera is pointed at section 100 of the industrial facility, the augmented reality device displays a graphical element (shown as graphical element 4 in the figure). This graphical element represents one or more parameters associated with the industrial equipment within the augmented reality device's view.

[0023] In one example, the augmented reality device includes software that can perform local SLAM (simultaneous localization and mapping) for the augmented reality. Specifically, the SLAM software can create a 3D SLAM map of the local optical features of the world around it and save the map to a server.

[0024] Furthermore, it can retrieve a map of pre-stored features from the server and then use that map for tracking. This means that the augmented reality device "knows" its own position and orientation within the (arbitrary) coordinate system of the SLAM map. The size of this map is limited to a specific 3D area, for example, approximately 10×10×10 meters.

[0025] In another example, the augmented reality device also includes GPS, Wi-Fi-based, or similar geolocation devices. This allows it to determine its location within a certain precision, such as about 5 meters outdoors or 50 meters indoors.

[0026] The augmented reality (AR) device connects to a server for displaying graphical elements. The server includes a map for locating the AR device, establishing its orientation and position relative to other equipment and gear within the industrial facility. The map is generated using a portable device and the server. The portable device has all the capabilities associated with the AR device and is able to maneuver itself around the industrial facility. In one example, the server stores information about the device and associated metadata in each subspace and can provide image processing capabilities such as image recognition, OCR, or 3D object detection. This will refer to... Figure 2 Further explanation.

[0027] Figure 2 A method 200 for generating maps for augmented reality devices in an industrial facility is illustrated. The industrial facility includes two or more subspaces. Here, a subspace refers to a logical portion or segment of an industrial facility that includes one or more industrial devices. Each subspace represents at least a portion of the physical infrastructure. Each subspace may also be referred to as an augmented reality bubble, i.e., a spatial region with defined constraints that forms an internal dimensional system. Although subspaces can be represented by arbitrary geometry, a spherical size can be chosen for simplicity. However, the terms "bubble" or "subspace" are not limited to a sphere. Alternatively, the linear geometry according to alternative embodiments may include straight edges to support modular stacking of subspaces within a mathematically higher coordinate system. According to an embodiment, the spherical size of the subspace may be selected as a diameter of 10 meters, surrounding a specific physical location, such as a specific outdoor GPS location, a specific indoor conference room, or a specific machine.

[0028] In one example, a model of an industrial facility is used to determine subspaces. Each of two or more subspaces includes multiple anchor points. Each of the multiple anchor points in a given subspace is fixed at a corresponding location within that subspace. In one example, the anchor points are fixed to industrial equipment within or near industrial equipment in the given subspace. For example, an anchor point is a radio frequency identification (RFID) tag fixed to a flow meter nameplate. Each anchor point has at least one anchor point identifier, which describes itself in such a way that it can be easily recognized by a user as a potential identifier and is easily recognized and interpreted by mobile augmented reality devices. The anchor point identifier has the property of relative uniqueness, meaning that it exists only once within the given subspace (or at least only once in a prominent location, or only a small number of times). Such anchor point identifiers can be, for example, the text of identifier labels in a physical plant. These are commonly already present in chemical, pharmaceutical, or power plants.

[0029] Furthermore, anchor identifiers can be, for example, text from street signs, posters, or other large signs; barcodes or QR codes from labels; object categories returned by image processing algorithms (e.g., "cactus" could be an identifier if there is only one cactus in the subspace); and / or object categories returned by 3D object detection algorithms (e.g., "pump type XYZ" could be an identifier if an algorithm exists capable of detecting and classifying all types of pumps in a processing plant). Information elements can also be referred to as holograms, which are understood below as containers used to precisely place and augment technical information relative to a real device (e.g., a component in physical infrastructure). Users can place, edit, modify, delete, and / or retrieve / view holograms. Holograms can be user-created annotations. These annotations can include speech (audio and speech-to-text); floating 3D models such as arrows; drawings; and photos and videos captured from devices or other files. Holograms can be, for example, text, 3D models, small animations, instruction documents, photos, videos, etc. Holograms can also contain links to live data, such as graphs from temperature sensors inside a machine. Or historical data. Each hologram has a 3D position in a coordinate system.

[0030] In one example, SLAM software on an augmented reality device can estimate the distance between identifiers (e.g., anchor identifiers and spatial environment identifiers) and the camera, thereby calculating the position of each identifier within the SLAM software's coordinate system or another such coordinate system.

[0031] Subspaces are created such that each subspace includes three or more anchor points, and each subspace overlaps with another subspace, with at least one anchor point present in the overlapping region. Therefore, the multiple anchor points of a given subspace include one or more anchor points located in the region overlapping with another subspace. This is in Figure 3 As shown in the image. Figure 3 A logical representation of a portion of an industrial facility comprising three subspaces 310, 320, and 330 is shown. Each subspace includes multiple anchor points. For example, subspace 310 includes anchor points 311, 313, 316, 319, 350, 360, and 363. Similarly, subspace 320 includes anchor points 321, 323, 326, 341, 343, and 339. Similarly, subspace 330 includes anchor points 336, 333, and 331. Furthermore, subspace 310 overlaps with subspaces 320 and 330. Therefore, anchor points 319 and 350 are located in the overlapping region between subspaces 310 and 320. Similarly, anchor points 350, 360, and 363 are located in the overlapping region between subspaces 310 and 330. Additionally, subspace 320 overlaps with subspace 330. Therefore, anchor points 339, 341, 343, and 350 exist in the overlapping region between subspaces 320 and 330. Anchor point 350 exists in the overlapping region shared by all three subspaces.

[0032] In one example, each subspace includes one or more spatial environment identifiers that can be read by an augmented reality device relative to the corresponding subspace in which the spatial environment identifier is located. For example, in an office building, "kitchen" could be such a spatial environment identifier for the subspace associated with the kitchen in the office building.

[0033] In step 210, the server and the portable device use a positioning subsystem to obtain location information of a first subspace among multiple subspaces. Here, the first subspace refers to the subspace in which the portable device currently exists. Therefore, in one example, to determine the first subspace, the portable device scans one or more anchor points within the first subspace and determines the first subspace in which the portable device exists based on the location of one or more anchor points. In another example, the portable device includes a positioning subsystem such as a Global Positioning System (GPS) and is capable of determining its location. Then, based on the location of the portable device, the first subspace in which the portable device exists is determined. In one example, referencing... Figure 3 The portable device exists in subspace 310. Therefore, the first subspace is subspace 310.

[0034] Then, in step 220, the server, together with the portable device, determines the coordinate system of the first subspace based on the obtained location information and multiple anchor points of the first subspace. In one example, referencing... Figure 3Based on the portable device's position within the first subspace, the portable device, together with the server, determines the coordinate system of subspace 310. Therefore, the coordinate system is used to determine the coordinates of each anchor point within the first subspace 310. The coordinate system mentioned here can refer to any well-known coordinate system, such as the Cartesian coordinate system, polar coordinate system, etc.

[0035] Then, in step 230, the server, together with the portable device, calculates the relative position and orientation of at least one adjacent subspace based on the determined coordinate system, for generating a map based on the position information of the first subspace and the relative positions of at least one adjacent subspace. As mentioned above, at least one adjacent subspace includes one or more anchor points in the region overlapping with the first subspace.

[0036] Continuing on Figure 3 In the example above, after generating the coordinate system of the first subspace, the coordinates of the anchor points relative to the coordinate system are calculated. This specifically includes the anchor points in the overlapping region of subspaces 320 and 330. Therefore, the coordinates of anchor points 319, 350, 360, and 363 are calculated using the coordinate system of the first subspace 310.

[0037] Then, based on the coordinates of the anchor points in the overlapping region, the server, together with the portable device, uses a coordinate system to calculate the anchor point coordinates in subspaces 320 and 330. Thus, the coordinate system of the first subspace is extended to the other subspaces. Therefore, by calculating the coordinates of the anchor points in subspaces 320 and 330, the server, together with the portable device, calculates the relative position and orientation of subspaces 320 and 330. This can be performed using various well-known transformation techniques. For example, techniques for best-fit rigid transformations for aligning two sets of corresponding points, such as those described by Hornung and Rabinovich of ETH Zurich in "Least Squares Rigid Motion Using SVD" in 2017, can be used. The calculated anchor point coordinates and the position and orientation information of the subspaces are then stored as a partial map of the industrial facility. This map can be used by augmented reality devices to improve the display of graphical elements. This is in Figure 4 The description will be further explained.

[0038] In one example, the coordinate system determined in method 200 above is further augmented with location information from a positioning system such as GPS. For example, the industrial facility is a portable facility (e.g., a portable or modular industrial system) and can be transported to different locations. Therefore, the coordinate system calculated above, along with the coordinates and locations, are reference values ​​before the industrial system is transported and when the industrial facility is installed at a specific location, with the location information at that specific location used to convert the map and coordinate system from relative values ​​to absolute values.

[0039] In one example, method 200 includes generating a graph associated with two or more subspaces. The graph includes multiple nodes and edges. Each of the multiple nodes is associated with one of the subspaces of the two or more subspaces and an anchor point of a first set of anchor points. Each anchor point of the first set of anchor points is located in at least one region overlapping with one of the two or more subspaces. Figure 5 An example of this type of diagram is shown in the figure.

[0040] Figure 5 It shows relative to Figure 3 Example graph 500 shows the subspace generation. Each subspace (310, 320, and 330) is shown as nodes (510, 540, and 580) in graph 500. Furthermore, each anchor point (319, 350, 360, 363, 341, 343, 339) in the overlapping region between subspaces 310, 320, and 330 is further shown as nodes (530, 555, 550, 560, 590, 570, 575) in graph 500. An edge from a subspace node to an anchor point node indicates that the corresponding anchor point is within the corresponding subspace. For example, subspace node 540 with an edge to anchor point node 550 indicates that anchor point 360 (represented by anchor point node 550) is within subspace 310 (represented by subspace node 540). After generating the graph, a subspace is selected as the initial subspace, and then subspaces adjacent to the first subspace within a specific radius are queried from the map.

[0041] Then, a depth-first search of the graph is performed, where the relative transformations of all subspaces relative to the first subspace are computed through successive transformation matrix multiplications. In one example, if a cyclic subgraph exists in the graph above, the above transformations are performed within a cycle, and the position of the initial bubble can be computed based on two other bubbles. This can be used to improve the relative positions of all bubbles in the cyclic graph by optimizing their respective origins to optimally align overlapping anchor points. If a bubble is part of two or more cyclic subgraphs, the optimization can be extended to the union of all subgraphs.

[0042] Figure 4 A method 400 for displaying data on an augmented reality device in an industrial facility is illustrated. As shown above, the industrial facility includes multiple subspaces (e.g., subspaces 310, 320, and 330).

[0043] In step 410, the augmented reality device determines a first subspace and its position and orientation within that subspace. The augmented reality device is located within the first subspace of the industrial facility. In one example, the augmented reality device identifies a first set of anchor points within the first subspace, which contains multiple subspaces. Each anchor point in the first set exists within the first subspace. Based on the identified anchor points in the first subspace, the augmented reality device determines its position and orientation within the first subspace. In another example, the augmented reality device uses one or more of its mapping sensors to generate a SLAM (Simultaneous Localization and Mapping) map to determine the first subspace and the device's orientation and position. In one example, the one or more mapping sensors include acoustic sensors, laser sensors, etc.

[0044] Then, in step 420, the augmented reality device determines one or more other subspaces based on its orientation in the first subspace, for displaying data associated with one or more objects in the one or more subspaces. To determine the one or more other spaces, the augmented reality device utilizes the map generated by the server described above. After determining the first subspace, the augmented reality device obtains the map from the server. Then, based on its position and orientation in the first subspace, the augmented reality device uses the map to determine its coordinates relative to the map coordinate system. The augmented reality device then determines the subspaces and anchor points that may be within the camera view of the augmented reality device based on its coordinates. As previously mentioned, each of the one or more other subspaces includes at least one anchor point in an area overlapping one or more of the first subspace and another subspace of the one or more other subspaces.

[0045] Then, in step 430, the augmented reality device populates its display with one or more data elements associated with one or more objects in the first subspace and one or more other subspaces. Based on the determined one or more other subspaces and the location and orientation of the augmented reality device, the augmented reality device determines the data elements of the objects in the first subspace and one or more subspaces. Then, based on the coordinates of the objects, the augmented reality device determines the location of the data elements on the display of the augmented reality device. Therefore, by using a map, the augmented reality device can effectively generate and locate data elements. The data elements displayed by the above method can help users perform actions in precise locations and increase the accuracy of these actions.

[0046] In one example, method 400 further includes assigning weights to anchor points and subspaces based on their distance from the augmented reality device. Therefore, for subspaces farther from the augmented reality device than a predetermined threshold distance, data elements associated with devices or industrial equipment in those subspaces are not displayed.

[0047] In one example, when two subspaces may not have anchor points in overlapping areas, the portable device, along with the server, can create anchor points such that they exist in both subspaces, or modify existing anchor points in one subspace to exist in both subspaces. For example, after determining the coordinate system of the first subspace, the portable device can approach an existing anchor point in the second subspace and use that coordinate system to calculate its position relative to the first subspace. Thus, the existing anchor point is included in the first subspace. While this is explained above with a single anchor point between two subspaces, the above aspects can be extended to modifying or adding multiple anchor points between multiple subspaces. The invention can take the form of a computer program product comprising a computer-usable or computer-readable medium storing program code that is used by or in connection with one or more computers, processing units, or instruction execution systems. For example, method 200 or method 400 can be implemented in a single device or across one or more devices.

[0048] Therefore, the present invention describes a server for generating maps for augmented reality devices in an industrial facility comprising two or more subspaces. The server includes one or more processors connected to a non-transient memory module comprising multiple instructions. The one or more processors of the server are configured to, upon execution of the instructions, obtain location information of a first subspace of the multiple subspaces, determine a coordinate system of the first subspace based on the obtained location information and multiple anchor points of the first subspace, and calculate the relative position and orientation of at least one adjacent subspace based on the determined coordinate system, for generating a map based on the location information of the first subspace and the relative position of at least one adjacent subspace, wherein the at least one adjacent subspace includes one or more anchor points in a region overlapping with the first subspace.

[0049] Additionally, the present invention describes an augmented reality device for displaying data associated with one or more objects in an industrial facility comprising multiple subspaces. The augmented reality device includes one or more processors connected to a non-transient memory module comprising multiple instructions. Upon execution of the instructions, the one or more processors are configured to determine a first subspace and the position and orientation of the augmented reality device within the first subspace, wherein the augmented reality device, within the first subspace of the industrial facility, determines one or more other subspaces based on the orientation of the augmented reality device in the first subspace, to display data associated with one or more objects in the one or more subspaces, wherein each of the one or more other subspaces includes at least one anchor point in a region overlapping one or more of the first subspace and another subspace of the one or more other subspaces, and populates the display of the augmented reality device with one or more data elements associated with one or more objects in the first subspace and the one or more other subspaces.

[0050] For the purposes of this specification, a computer-usable or computer-readable non-transient storage medium can be any means capable of containing, storing, transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. Such a medium can be electronic, magnetic, optical, electromagnetic, infrared, or a semiconductor system (or apparatus or device) or its own propagation medium, as signal carriers are not included in the definition of a physical computer-readable medium, which includes semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), rigid disks, and optical disks, such as optical disc read-only memory (CD-ROM), optical disc read / write, DVD, and Blu-ray. The processing units and program code used to implement each aspect of this technology can be centralized or distributed (or a combination thereof) as known to those skilled in the art.

[0051] Although the invention has been described with reference to a few industrial devices, many industrial devices may be utilized in the context of the invention. While the invention has been described in detail with reference to certain embodiments, it should be understood that the invention is not limited to those embodiments. In view of the invention, many modifications and variations will be apparent to those skilled in the art without departing from the scope of the various embodiments of the invention as described herein. Therefore, the scope of the invention is defined by the appended claims rather than by the foregoing description. All changes, modifications, and variations within the meaning and scope of the equivalents of the claims are considered to be within its scope. All advantageous embodiments claimed in the method claims can also be applied to the device / non-transient storage medium claims.

Claims

1. A method (200) for generating maps for augmented reality devices in an industrial facility comprising two or more subspaces (310, 320, 330), each of the two or more subspaces (310, 320, 330) comprising a plurality of anchor points (311, 313, 316, 319, 350, 360, 363, 321, 323, 326, 341, 343, 339, 336, 333, 331), wherein, The plurality of anchor points in the corresponding subspace includes one or more anchor points (350, 360, 363, 319, 341, 343, 339) located in a region overlapping with another subspace in the plurality of subspaces (310, 320, 330), and the method (200) includes: a. Obtain the position information of the first subspace (310) among multiple subspaces (310, 320, 330) (210); b. Based on the obtained position information of the first subspace (310) and multiple anchor points (319, 350, 360, 363, 316, 313, 311), determine (220) the coordinate system of the first subspace (310); and c. Based on the determined coordinate system, calculate (230) the relative position and orientation of at least one adjacent subspace (330) for generating the map based on the position information of the first subspace (310) and the relative position of the at least one adjacent subspace (330), wherein the at least one adjacent subspace (330) includes one or more anchor points (350, 360, 363) in the region overlapping with the first subspace (310). Calculating the relative positions and orientations of the at least one adjacent subspace (330) further includes generating a graph (500) associated with the two or more subspaces (310, 320, 330), wherein the graph (500) includes a plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) and edges, the plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) Each node in the list is associated with one of the two or more subspaces (310, 320, 330) and an anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339), wherein each anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339) is located in at least one region that overlaps with one of the two or more subspaces (310, 320, 330) and another subspace. Select the first subspace (310) of the two or more subspaces (310, 320, 330) as the initial subspace, and Perform a depth-first search of the graph (500), wherein performing the depth-first search includes using one or more transformation matrix multiplications to compute the relative transformation of one or more subspaces (320, 330) with respect to the first subspace (310).

2. A method (400) for displaying data on an augmented reality device in an industrial facility comprising multiple subspaces (310, 320, 330), the method (400) comprising: a. Determine (410) the first subspace (310) and the position and orientation of the augmented reality device in the first subspace (310), wherein the augmented reality device is located within the first subspace (310) of the industrial facility; b. Based on the orientation of the augmented reality device in the first subspace (310), determine one or more other subspaces (320, 330) to display data associated with one or more objects in the one or more subspaces (320, 330), wherein each of the one or more other subspaces (320, 330) includes at least one anchor point in a region that overlaps with one or more of the first subspace (310) and another subspace of the one or more other subspaces (320, 330); c. Populate the display of the augmented reality device with one or more data elements associated with one or more objects in the first subspace (310) and the one or more other subspaces (320, 330); and Generate a graph (500) associated with the plurality of subspaces (310, 320, 330), wherein the graph (500) includes a plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) and edges, and each of the plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) is associated with the plurality of subspaces. A subspace (310, 320, 330) and an anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339) are associated, wherein each anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339) is located in at least one region that overlaps with one subspace and another subspace of the plurality of subspaces (310, 320, 330). The first subspace (310) among the plurality of subspaces (310, 320, 330) is selected as the initial subspace, and Perform a depth-first search of the graph (500), wherein performing the depth-first search includes using one or more transformation matrix multiplications to compute the relative transformation of one or more subspaces (320, 330) with respect to the first subspace (310).

3. The method (400) according to claim 2, wherein, The method (400) further includes: determining anchor points in the first subspace (310) for determining the orientation and position of the first subspace (310) and the augmented reality device in the first subspace (310), wherein each anchor point in the first subspace is a physical identifier attached to a corresponding position within the first subspace (310).

4. The method (400) according to claim 3, wherein, The method (400) further includes generating a map using one or more mapping sensors to determine the orientation and location of the first subspace (310) and the augmented reality device.

5. A server for generating maps for augmented reality devices in an industrial facility comprising two or more subspaces (310, 320, 330), each of the two or more subspaces (310, 320, 330) comprising a plurality of anchor points (311, 313, 316, 319, 350, 360, 363, 321, 323, 326, 341, 343, 339, 336, 333, 331), wherein, The plurality of anchor points in the corresponding subspace includes one or more anchor points located in a region that overlaps with another subspace in the plurality of subspaces (310, 320, 330); The server includes: a. One or more processors connected to the non-transient memory module, said one or more processors being configured to: i. Obtain the position information of the first subspace (310) among multiple subspaces (310, 320, 330); ii. Determine the coordinate system of the first subspace (310) based on the obtained position information of the first subspace (310) and multiple anchor points; iii. Based on the determined coordinate system, calculate the relative position and orientation of at least one adjacent subspace (330) for generating the map based on the position information of the first subspace (310) and the relative position of the at least one adjacent subspace (330), wherein the at least one adjacent subspace (330) includes one or more anchor points in the region overlapping with the first subspace (310); and Generate a graph (500) associated with the two or more subspaces (310, 320, 330), wherein the graph (500) includes a plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) and edges, each of the plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) being associated with the two or more subspaces (310, 320, 330), and edges. A subspace within the space (310, 320, 330) is associated with an anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339), wherein each anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339) is located in at least one region overlapping with one subspace and another subspace of the two or more subspaces (310, 320, 330). Select the first subspace (310) of the two or more subspaces (310, 320, 330) as the initial subspace, and Perform a depth-first search of the graph (500), wherein performing the depth-first search includes using one or more transformation matrix multiplications to compute the relative transformation of one or more subspaces (320, 330) with respect to the first subspace (310).

6. A non-transient storage medium for generating maps for augmented reality devices in an industrial facility comprising two or more subspaces (310, 320, 330), each of the two or more subspaces (310, 320, 330) comprising a plurality of anchor points, wherein, The plurality of anchor points in the corresponding subspaces include one or more anchor points located in a region that overlaps with another subspace in the two or more subspaces (310, 320, 330); Includes multiple instructions, which, when executed on one or more processors, cause the one or more processors to: a. Obtain the position information of the first subspace (310) among multiple subspaces (310, 320, 330); b. Determine the coordinate system of the first subspace (310) based on the obtained position information and multiple anchor points; c. Based on the determined coordinate system, calculate the relative position and orientation of at least one adjacent subspace (330) for generating the map based on the position information of the first subspace (310) and the relative position of the at least one adjacent subspace (330), wherein the at least one adjacent subspace (330) includes one or more anchor points in the region overlapping with the first subspace (310); and Generate a graph (500) associated with the two or more subspaces (310, 320, 330), wherein the graph (500) includes a plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) and edges, each of the plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) being associated with the two or more subspaces (310, 320, 330), and edges. A subspace within the space (310, 320, 330) is associated with an anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339), wherein each anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339) is located in at least one region overlapping with one subspace and another subspace of the two or more subspaces (310, 320, 330). Select the first subspace (310) of the two or more subspaces (310, 320, 330) as the initial subspace, and Perform a depth-first search of the graph (500), wherein performing the depth-first search includes using one or more transformation matrix multiplications to compute the relative transformation of one or more subspaces (320, 330) with respect to the first subspace (310).

7. An augmented reality device for displaying data associated with one or more objects in an industrial facility comprising multiple subspaces, the augmented reality device comprising: a. One or more processors connected to the non-transient memory module, said one or more processors being configured to: i. Determine a first subspace and the position and orientation of the augmented reality device within the first subspace, wherein the augmented reality device is located within the first subspace of the industrial facility; ii. Determine one or more other subspaces based on the orientation of the augmented reality device in the first subspace to display data associated with one or more objects in the one or more subspaces, wherein each of the one or more other subspaces includes at least one anchor point in a region that overlaps with one or more of the first subspace and another subspace in the one or more other subspaces; iii. Populate the display of the augmented reality device with one or more data elements associated with one or more objects in the first subspace and the one or more other subspaces; and Generate a graph (500) associated with the plurality of subspaces (310, 320, 330), wherein the graph (500) includes a plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) and edges, and each of the plurality of nodes (510, 530, 540, 550, 560, 570, 575, 580, 590, 555) is associated with the plurality of subspaces. A subspace (310, 320, 330) and an anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339) are associated, wherein each anchor point in the first set of anchor points (319, 350, 360, 363, 341, 343, 339) is located in at least one region that overlaps with one subspace and another subspace of the plurality of subspaces (310, 320, 330). The first subspace (310) among the plurality of subspaces (310, 320, 330) is selected as the initial subspace, and Perform a depth-first search of the graph (500), wherein performing the depth-first search includes using one or more transformation matrix multiplications to compute the relative transformation of one or more subspaces (320, 330) with respect to the first subspace (310).

8. A non-transient storage medium for displaying data associated with one or more objects in an industrial facility comprising multiple subspaces, the non-transient storage medium comprising a plurality of instructions, which, when executed on one or more processors, cause the one or more processors to: a. Determine the first subspace and the position and orientation of the augmented reality device within the first subspace, wherein, The augmented reality device is located in the first subspace of the industrial facility; b. Determine one or more other subspaces based on the orientation of the augmented reality device in the first subspace to display data associated with one or more objects in the one or more subspaces, wherein each of the one or more other subspaces includes at least one anchor point in a region that overlaps with one or more of the first subspace and another subspace in the one or more other subspaces; c. Populate the display of the augmented reality device with one or more data elements associated with one or more objects in the first subspace and the one or more other subspaces; and Generate a graph associated with the plurality of subspaces, wherein the graph includes a plurality of nodes and edges, each of the plurality of nodes being associated with one of the plurality of subspaces and an anchor point in a first set of anchor points, wherein each anchor point in the first set of anchor points is located in at least one region overlapping with one of the plurality of subspaces and another subspace. The first subspace among the plurality of subspaces is selected as the initial subspace, and Perform a depth-first search on the graph, wherein performing the depth-first search includes using one or more transformation matrix multiplications to compute the relative transformation of one or more subspaces with respect to the first subspace.

Images