3D mapping of process control environments

Abstract

To provide an augmented reality platform suitable for use in process control environment.SOLUTION: In a method of mapping real-world process control environment, a mobile device is registered (602) at a reference location, and 3D positions and orientations of the mobile device are tracked (604) by using an inertial measurement unit. User input indicating that a new node is to be added to a 3D map of the process control environment is detected (606), and a 3D position of a real-world object relative to the reference location is determined, or caused to be determined (608), on the basis of a tracked 3D position and an orientation of the mobile device. A node database is caused to add (610) a new node to the 3D map of the process control environment, at least by causing the 3D position of the real-world object to be stored in association with the new node.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] This disclosure generally relates to augmented reality technology, and more particularly to augmented reality platforms that can be used in process control or other environments.

Background Art

[0002] Augmented reality (AR) systems that provide an overlay of digital information (such as text, images, animations, etc.) in a real-world environment are becoming increasingly common as new uses for this technology are considered. AR systems typically provide information specific to the context of the user's current real-world environment. In such systems, knowledge of the real world, such as knowledge of physical objects currently in the user's view, and the physical layout of the user's current location and its surroundings, is generally required. To make this type of information available to an AR system, a three-dimensional (3D) digital model representing the user's environment and the objects within it is typically created. The 3D model can then be augmented with specific types of data (such as descriptions), graphics (such as icons), etc. associated with specific locations and / or specific objects within the environment. When a user equipped with appropriate AR gear moves through the mapped environment, the augmented portions can be rendered / displayed to the user by overlaying digital information on the user's real-world view (such as a real-time camera view, or a head-up display (HUD) through which the user observes the environment), without rendering / displaying the remaining portions of the 3D model.

[0003] Traditionally, this type of 3D model has been manually constructed by human designers and / or using 3D scanners. However, both of these approaches can be time-consuming and labor-intensive for applications involving environments with complex layouts and / or a large number of objects. Furthermore, changes to the environment (e.g., objects being moved or replaced / updated within the environment) may require the entire modeling process to be repeated or other time-consuming and labor-intensive steps.

[0004] Current AR systems also suffer from other drawbacks. For example, when used in relatively dense environments (e.g., environments with multiple objects that are close together and / or have a large amount of information to display in the AR view), AR systems can easily overwhelm the user with a vast amount of information, and / or the user may not be able to easily perceive the context of all the information. Furthermore, some objects may become obscured within the AR view. For example, objects may be located within an enclosure or in an inaccessible or inconvenient (e.g., remote) location. [Overview of the project]

[0005] This specification discloses technologies, systems, apparatus and methods for providing augmented reality experiences. These technologies, systems, apparatus and methods may be applied, for example, to industrial process control systems, environments and / or plants, which may be synonymously referred to herein as “process control” or “process” systems, environments and / or plants. Typically, such systems and plants provide control of one or more processes that generate or produce products by manufacturing, purifying or otherwise transforming raw physical material in a distributed manner. However, in some embodiments, the technologies, systems, apparatus and methods described herein may be used in other types of environments and / or in contexts other than augmented reality (e.g., virtual reality (VR) contexts).

[0006] In general, process control environments can present many of the various challenges associated with the creation and use of conventional augmented reality (AR) systems or platforms, as outlined above in the "Background Technology" section. For example, such environments often contain a large number of diverse objects (e.g., tanks, pumps, valves, motors, transmitters, etc.) that are in close proximity and / or associated with a large amount of information (e.g., sensor readings, diagnostic status, etc.). Furthermore, these objects may be frequently moved or replaced.

[0007] The AR platforms described herein may overcome or improve upon some or all of these difficulties and may therefore be well suited for use in process control environments. In the AR platform, three-dimensional (3D) models are generated by the user using an AR mobile device. The AR mobile device may be an AR helmet, AR headset, AR goggles, smart glasses, smartphone, tablet, or any other suitable type of mobile device (as described below) that can support location tracking and present the user with an AR experience.

[0008] A 3D model shows the 3D location / position of a point of interest within a process control environment, corresponding to different real-world objects (e.g., field devices, tanks, conveyors, controllers, switches, etc.). Objects added to the 3D model are referred to herein as “nodes” of the AR platform or system. As used herein, the term “object” (also referred to as “real-world object”) may be any physical thing (e.g., a device, component, equipment, structure, etc.), a part of something (e.g., an element of a device or component), or a combination of things (e.g., a group of related devices or components, etc.). In a process control environment, for example, an object may be a field device (e.g., a valve, pump, sensor, etc.), a part of equipment (e.g., a conveyor, tank, etc.), a process controller, a switch, etc.

[0009] In some implementations, when an AR application is launched on an AR mobile device (for example, when it is specifically launched by a user or when the device is powered up), the user "checks in" or registers the AR mobile device at some criterion or "landmark" location. For example, the user may use the AR mobile device's camera to scan a QR code or another visual representation of a code / identifier at a landmark location. The act of scanning the code may establish the user / device's starting position (e.g., coordinates {0,0,0} in {x,y,z} space).

[0010] After registration, the user may begin to physically move through the process control environment using an AR mobile device. The user's movement through the process plant may be tracked using appropriate positioning techniques (e.g., provided by the user's AR mobile device). Other positioning and / or orientation techniques may be used, as GPS or GNSS units may not function in many process plant environments and may not be able to provide orientation data or accurate elevation data (especially in indoor process control environments). For example, the position of the AR mobile device relative to the location of landmarks, and the orientation of the AR mobile device (e.g., the direction of the AR mobile device's camera's field of view, which may correspond to the direction the user is facing), may be tracked using fusion of inertial measurement unit (IMU) data (e.g., generated by an accelerometer, gyroscope, etc.) and camera image data. In one implementation, position and orientation may be tracked using, for example, the Visual Inertial Odometry (VIO) technique developed by Qualcomm®.

[0011] When a user approaches a real-world object to be added as a node, or is near one, and is facing the object (e.g., wearing an AR helmet or goggles) or pointing the device camera at the object (e.g., if the device is a tablet or smartphone), the user may choose to add a new node. For example, an AR mobile device may recognize the voice command “Add Node” when uttered by the user, or (e.g., if the device is a tablet or smartphone) the AR mobile device may allow the user to select “Add Node” or a similar option / control on the graphical user interface (GUI) of the AR mobile device display. The AR mobile device may then capture an image of the object and process the image to estimate the distance between the AR mobile device and the object. Alternatively, other sensors (e.g., Lidar, radar, etc.) may be used to determine the distance between the AR mobile device and the object. The AR mobile device may then determine the object’s position relative to a landmark location based on the AR mobile device’s current position and orientation, as well as the distance between the AR mobile device and the object. Alternatively, a remote server or other computing device / system may process image data captured by the AR mobile device to determine the object's location relative to a landmark.

[0012] The user may also specify the name or type of a new node (e.g., valve, pump, etc.) and / or other information (e.g., the object's model number or other identifier, a short description of the object, etc.). The specified information can then be associated with the new node in the AR system's backend database. The backend database contains a library of nodes added by the user (and possibly other users) that have 3D locations associated with landmark locations, and can collectively establish a virtual 3D "map" of objects / assets within the process control environment. The database may also store other node information, such as a list of associated and / or connected nodes.

[0013] After at least one object of interest within a particular process control environment has been mapped (i.e., added to the node library using an AR mobile device), any user (including the mapping user) who registers their AR mobile device at a landmark location may take an extended tour of the process control environment. When a user with a registered AR mobile device directs the viewfinder of the AR mobile device's camera towards an area containing a particular real-world object (for example, when the user views the area while wearing an AR helmet or goggles), and the object is already associated with a node in the node library, the AR mobile device may extend the real-world scene observed by the user with node "markers" (e.g., text, icons, graphics, etc.) that appear to the user as if they were located at or near the coordinates of the object in the user's real-world view.

[0014] When a user selects a particular node (for example, by focusing on a node marker or issuing a voice command), the AR mobile device may make one or more “node experiences” available to the user. For example, one node experience may overlay text (node ​​name, description, etc.), tables, and / or graphics / icons onto the user’s real-world view. Other node experiences may overlay web content from a specific URL (e.g., an instructional or “help” video), or, for example, a visualization of physical or logical connections to other objects / nodes and / or their relationships. Still other node experiences may include searching for work orders associated with an object / node, establishing a communication link between a remotely located user and the appropriate expert, etc. In some implementations, at least one node body The experience launches (or links to) one or more other software applications or systems in a specific context of an object being observed by the user. For example, an AR mobile device may overlay sensor data, state, and / or virtual controls of a selected object / node (e.g., a field device) within its field of view, with data received from different applications (and user input to the controls sent to different applications).

[0015] The experiences available to a user for a given node, the types of content presented within a particular node experience, and / or, in some implementations, whether any extended information is available to a particular user for a given node, can be determined at least partially based on the user's "role." For example, a particular user may use an AR mobile device to select, or be pre-assigned to, an "operator," "maintainer," or "engineer" role.

[0016] Many other features can be offered to the user. In some implementations, a virtual "X-ray vision" can be provided to the user by creating a surreal or pseudo-realistic visualization of a 3D model that, when superimposed on the user's view of the real world, may appear as if the user is looking at an object hidden inside an enclosure (e.g., inside a cabinet) or an object behind a barrier. AR mobile devices can also present (and / or allow the user to choose) specific node experiences associated with the hidden object, such as displaying state data or alerts associated with the hidden object, or issuing voice commands to control the hidden object's behavior.

[0017] As another example, an AR mobile device could overlay “avatars” of restricted-access and / or remote objects / nodes onto the user’s real-world view. These objects could be, for example, the top of a tall tower or a toxic or high-voltage area. Similar to hidden objects, users could be provided with specific node experiences that offer various interactions with restricted-access and / or remote objects. In both hidden object (“X-ray vision”) and restricted-access / remote object implementations / scenarios, users could be provided with an overall experience very similar to having the object / node directly in front of them and being clearly visible to them while the user interacts with it.

[0018] Other types of functionality may also be supported by the AR platform. For example, a user's current location could trigger the display of various alerts (e.g., warnings that the user is in a dangerous area) and / or other information. As another example, a user's current and / or past 3D location within a process plant could be tracked / recorded for various purposes, such as ensuring employee evacuation in an emergency, ensuring employees are not injured or require medical assistance, or monitoring the training of specific employees.

[0019] The AR platform described above and further below may offer several advantages depending on the specific implementation. For example, by mapping only the relative 3D position of a particular object, a 3D model can be generated more easily and efficiently than by conventional means (e.g., mapping the environment using a 3D scanner). Similarly, updating or reconfiguring objects within a process-controlled environment can be performed more quickly and easily. Furthermore, by using the mapping process described herein and by tracking the movement of the AR mobile device / user as described herein, the AR platform can be implemented without requiring cost and / or retrofit time, or otherwise with short-range communication technology (e.g., Bluetooth or Near Field Communication (NFC) unit). Without requiring the user to equip objects and without requiring time-consuming actions by the user navigating the environment (e.g., scanning QR codes on objects, manually entering object identifiers, connecting to physical objects), it is possible to determine when to provide the user with augmented information. Therefore, traditional barriers to the deployment of AR systems, particularly in process control environments, can be mitigated or eliminated.

[0020] As another example, an AR platform may generally provide a marketplace where third parties can propose new, customized types or classes of “node experiences” that correspond to different user interactions with objects / nodes, and / or propose new content for existing node experiences. Instrumentation vendors or other entities may, for example, propose subscriptions to “help” information for specific objects, or AR or video-based work instructions. In some embodiments, a virtual “app store” may allow customers to license various applications and / or subscribe to specific types of content desired for their mobile workers.

[0021] Furthermore, features that facilitate the display / viewing of extended information for obscure, remote, and / or restricted access nodes can save users time (and therefore cost), and in some implementation forms and / or scenarios, can improve worker safety by avoiding the need to visit hazardous areas.

[0022] Furthermore, the AR platform architecture may be highly flexible. An AR platform can facilitate the display of relevant information to users on various different platforms, such as smart helmets, smart goggles, smartphones, and tablets. An AR platform can also facilitate interaction with numerous other software applications (e.g., process control software that can retrieve data indicating control logic, device readings, etc.) and / or new experiences, content, or other functions that should be easily provided.

[0023] As stated above, the specific implementations and / or features described herein do not need to be used in AR systems and / or process control environments. For example, a “node experience” based on the mapped 3D positions of objects in a plant, as well as the user’s current position and orientation, can be represented to a user with a smartphone, tablet, or other mobile device without necessarily having to be provided as an AR display (e.g., using a virtual reality display or simply a GUI). As another example, the technologies described herein may be applied to environments other than process control environments, such as office spaces, warehouses, and hospitals. [Brief explanation of the drawing]

[0024] [Figure 1] This is a block diagram illustrating an example of an augmented reality (AR) system that provides an AR platform capable of supporting the various methods, techniques, functions, and / or features described herein. [Figure 2]FIG. 1 is a block diagram depicting components of an exemplary process control environment in which an AR system can be utilized. [Figure 3] Draws a three-dimensional (3D) map of an exemplary node in a process control environment. [Figure 4A] Draws an exemplary user interface that can be superimposed over the view provided by an AR mobile device. [Figure 4B] Draws an exemplary user interface that can be superimposed over the view provided by an AR mobile device. [Figure 4C] Draws an exemplary user interface that can be superimposed over the view provided by an AR mobile device. [Figure 4D] Draws an exemplary user interface that can be superimposed over the view provided by an AR mobile device. [Figure 4E] Draws an exemplary user interface that can be superimposed over the view provided by an AR mobile device. [Figure 4F] Draws an exemplary user interface that can be superimposed over the view provided by an AR mobile device. [Figure 4G] Draws an exemplary user interface that can be superimposed over the view provided by an AR mobile device. [Figure 4H] Draws an exemplary user interface that can be superimposed over the view provided by an AR mobile device. [Figure 5A] Shows elements of an exemplary database that can be used in an AR platform. [Figure 5B] Shows elements of an exemplary database that can be used in an AR platform. [Figure 6] Draws an exemplary embodiment and scenario in which a user of an AR mobile device is provided with virtual X-ray vision. [Figure 7A]To provide virtual proximity between the user and objects that are far away from the user or otherwise difficult to access, an exemplary user interface is drawn that can be overlaid on the view provided by the AR mobile device. [Figure 7B] To provide virtual proximity between the user and objects that are far away from the user or otherwise difficult to access, an exemplary user interface is drawn that can be overlaid on the view provided by the AR mobile device. [Figure 8] This is a flowchart illustrating an exemplary method for mapping a real-world process control environment using a mobile device. [Figure 9] This is a flowchart illustrating an exemplary method for providing virtual enhanced vision to users of AR mobile devices in a real-world process control environment. [Figure 10] This is a flowchart illustrating an exemplary method for facilitating interaction between users of AR mobile devices and real-world objects that may be present in remote or restricted access areas of a process control environment. [Modes for carrying out the invention]

[0025] Exemplary Augmented Reality System Figure 1 illustrates an example of an augmented reality (AR) system 10 that may generally provide an AR platform for delivering AR experiences in an environment (e.g., a process control environment). The AR system 10 includes a user's AR mobile device 12 and a backend server 14 that can be communicatively coupled to the AR mobile device 12 via a network 16. The AR mobile device 12 may be any computing device that has appropriate processing and sensing capabilities and is wearable or otherwise portable by the user. For example, the AR mobile device 12 may be a device specifically configured for AR, such as an AR helmet (e.g., Smart Helmet® developed by DAQRI®) or AR goggles. Alternatively, the AR mobile device 12 may also be a device with non-AR applications (e.g., a tablet, smartphone, smart glasses, smartwatch, etc.) but runs software applications that configure the device 12 to have AR capabilities. Although Figure 1 depicts only one AR mobile device 12, many more AR mobile devices, identical or different from the AR mobile device 12, could be used in a similar manner in the AR system 10. It is understood that this is the case.

[0026] The AR mobile device 12 is generally configured to provide the user with an AR experience by augmenting the user's view of the real world with contextual information (e.g., text, graphics, animations, etc.). The backend server 14 generally supports the AR experience of the user and other users of AR mobile devices by receiving data from the AR mobile device that indicates the current state and / or environment of the AR device and / or the user, and by providing data to the AR mobile device as needed, by managing data that specifies how the user's view of the real world should be augmented in particular situations.

[0027] Network 16 includes one or more wireless networks, and possibly one or more wired networks. In the exemplary embodiment of Figure 1, the AR platform of AR system 10 utilizes a cloud-based architecture, and network 16 includes the internet. If AR system 10 is used indoors, network 16 may also include wireless networks that do not require any direct communication with remote towers or base stations, such as IEEE 802.11 or "Wi-Fi" networks. However, in other embodiments, network 16 includes cellular networks (e.g., LTE, GSM, etc.). As will be further described below, AR system 10 may also include one or more other servers 18 that are communicatively coupled to the backend server 14 and / or AR mobile device 12 and collectively store and run one or more applications 19.

[0028] As shown in the example of the embodiment in Figure 1, the AR mobile device 12 may include a network interface 20, a display 22, a camera 24, an inertial measurement unit (IMU) 26, and memory 30 for storing an AR application 32. The network interface 20 is configured to enable communication with remote computing devices and systems, including a backend server 14, using at least some wireless communication protocols of the network 16 (e.g., Wi-Fi or a cellular network).

[0029] The display 22 may include hardware and associated firmware and / or software configured according to any suitable type of digital display technology. For example, the display 22 may use liquid crystal display (LCD) technology, light-emitting diode (LED) technology, organic light-emitting diode (OLED) technology, etc. The display 22 may generally be transparent, translucent, or opaque. The structure or form factor of the display 22, and whether the display is transparent / translucent or opaque, generally depends on the type of AR mobile device 12. For example, if the AR mobile device 12 is a helmet, the display 22 may have the form of a visor and may be translucent so that any information displayed by the display 22 is superimposed on the user's direct view of the real world (i.e., a "head-up display" or "HUD"). Conversely, if the AR mobile device 12 is a tablet or smartphone, the display 22 may have a conventional rectangular form factor and may only allow an indirect view of the real world (e.g., as captured by a camera 24).

[0030] The camera 24 may be any suitable type of optical sensor, such as a charge-coupled device (CCD) image sensor. In alternative embodiments, the camera 24 may be a substitute for or include a different type of imaging device, such as a light-detecting and ranging (lidar) sensor or a radar sensor. The IMU 26 may include one or more sensors (e.g., an accelerometer and / or gyroscope) that generate data indicating the movement of the AR mobile device 12 in three dimensions. Although not shown in Figure 1, the AR mobile device may also have other components, such as a microphone and / or speaker.

[0031] The memory 30 that stores the AR application 32 may be any suitable type of persistent memory, such as read-only memory (ROM) in solid-state or hard drive format. The AR application 32 generally adjusts the AR experience of the user of the AR mobile device 12 by, for example, generating appropriate augmentation information on the display 22 and communicating with the backend server 14 as needed. Figure 1 shows an exemplary set of modules corresponding to functions or features that may be performed or provided by the AR application 32, including a role identification module 40, a registration module 42, a positioning and orientation module 44, a node experience module 46, a node creation module 48, a visual enhancement module 50, and a location-based alert module 52. Each of modules 40-52 is described further below in relation to the operation of the AR system 10. In some embodiments, the AR application 32 includes fewer, more, and / or different modules than those shown in Figure 1. For example, the AR application 32 may include a speech recognition module for recognizing user voice commands.

[0032] In the exemplary embodiment shown in Figure 1, the backend server 14 also includes a web service interface 60, an experience provisioning unit 62, a 3D model generation unit 64, a visual enhancement unit 66, an alert generation unit 68, and a user monitoring unit 70. Generally, the web service interface 60 may provide an object-oriented web-based interface to an AR database 72 coupled to (or contained within) the backend server 14. In some embodiments, the web service interface 60 provides an application programming interface (API) to a remote device such as an AR mobile device 12.

[0033] The AR database 72 generally stores information that defines a 3D map of a particular environment (for example, by storing node identifiers, the location of nodes in the 3D map relative to landmarks, and possibly other information associated with the nodes), as described below in more detail in relation to Figures 5A and 5B (according to a particular embodiment). The AR database 72 may be a single database or a collection of databases, may be stored in a single physical memory, or may be distributed across multiple memories at one or more geographic locations. One, some, or all of the elements / units 60-70 may be implemented as software instructions stored in persistent memory (e.g., ROM). In other embodiments, the backend server 14 may contain fewer, more, and / or different elements / units than those shown in Figure 1.

[0034] While in operation, a user wearing or otherwise carrying the AR mobile device 12 can launch the AR application 32 on the AR mobile device 12. The AR application 32 can be launched, for example, by manually selecting an icon on the display 22, issuing a voice command, or simply by powering up the AR mobile device 12.

[0035] In some embodiments, the role identification module 40 of the AR application 32 initially prompts the user to indicate a specific “role” that may correspond to the user’s job location and / or a particular task or task that the user is trying to accomplish, for example, by using the AR system 10. In other embodiments, the user is initially prompted to indicate their role later (for example, after registering with a landmark as described below). An example of a user interface that the role identification module 40 may overlay on or through the display 22 onto a real-world view seen (for example, on an image frame captured by the camera 24, or on the user’s direct real-world view) is described below in relation to Figure 4A. In alternative embodiments, Rather than prompting the user to select a role, the role identification module 40 determines the user's pre-assigned role based on the identifier and / or identification information (e.g., name, employee number, etc.) of the AR mobile device 12 entered or spoken by the user.

[0036] The selected or assigned user roles can tailor various aspects of the AR experience provided to the user, as will be further described below. Depending on the embodiment, the user role may be fixed after being selected or assigned, or it may be changed on the fly by the user as they move through the environment. In some embodiments, the role identification module 40 is not included in the AR application 32, and no user roles are selected or assigned within the AR system 10.

[0037] Before mapping an environment, or before updating or exploring a previously mapped environment, in some embodiments, the user may need to register the AR mobile device 12 with a “landmark” whose location will then be used as a reference point for future movement of the AR mobile device 12 (and, similarly, the user). The same landmark location is also used as a reference point for the location of any object in the environment that is already established (or will be established) as a node in the 3D map, thereby enabling the determination of the location of the AR mobile device 12 relative to any mapped object. The landmark location may represent, for example, {0,0,0} in a {x,y,z} coordinate system, or another coordinate system (e.g., polar coordinates) may be used.

[0038] To register the AR mobile device 12, the registration module 42 may process one or more images / frames captured by the camera 24. The landmark may be, for example, a QR code, or any other suitable type of image or text physically printed at the landmark location (e.g., a wall or door near the entrance to an area within a process control plant). In some embodiments, the registration module 42 or another part of the AR application 32 may overlay a reticle within the real-world view provided to the user (e.g., as described later in relation to Figure 4E) to help the user focus on the QR code, image, etc. In some embodiments, different techniques are used to recognize the landmark, such as a complementary Bluetooth or NFC communication unit of the AR mobile device 12 and another device fixed at the landmark location.

[0039] In some embodiments, when a user registers an AR mobile device 24 with a specific landmark, the registration module 42 transfers data indicating the landmark identifier to the backend server 14 via the network 16 and the web service interface 60 (for example, by sending a binary representation of a QR code or by sending an image of a QR code before decrypting the image). The backend server 14 may then compare the landmark identifier with a landmark identifier stored in the AR database 72 (if one exists). If the identifier is not yet stored, the backend server 14 may return a failure message to the registration module via the web service interface 60 and the network 16, or, in some embodiments, may cause the AR database to create a new landmark identifier input associated with a new 3D map. Conversely, if the landmark identifier already exists in the AR database 72, the backend server 14 may make some or all of the corresponding 3D map (and associated data) available to the AR mobile device 12 and return a message indicating successful registration to the registration module 42 via the web service interface 60 and the network 16.

[0040] The AR database 72 can store a large number of different landmark identifiers, each associated with a different 3D map, and each map is associated with a library of different nodes. Multiple maps can be associated with the same environment (for example, by adding different objects within a single environment as nodes in different maps), and / or different maps can be associated with different environments (for example, a map for the first area of ​​a process control plant, another map for the second area of ​​the plant, etc.).

[0041] In some embodiments, successfully registering the AR mobile device 12 with a landmark causes the positioning and orientation module 44 to set the position of the AR mobile device 12 equal to the landmark position (e.g., {0, 0, 0}). However, for greater accuracy, the registration module 42 may use depth sensing techniques to process the image of the landmark captured by the camera 24 to determine the distance between the AR mobile device 12 and the landmark. The positioning and orientation module 44 may then offset the starting position of the AR mobile device 12 from the known / reference position of the landmark based on both the determined distance from the landmark and the orientation of the AR mobile device 12 relative to the landmark. If the landmark is a QR code or other visual identifier, the relative orientation of the AR mobile device 12 may be assumed based on the direction the landmark is facing. Alternatively, the relative orientation may be determined from the captured image (e.g., by determining the angle of incidence of the camera view relative to the landmark).

[0042] Once registered, the user of the AR mobile device 12 can begin navigating the environment using a real-world view (provided via the display 22) that is generally augmented by digital text, tables, graphics, animations, and / or other types of information associated with 3D maps corresponding to landmarks. To track the user's (i.e., the AR mobile device 12's) position and orientation, the positioning and orientation module 44 may access the IMU 26, the camera 24, and / or one or more other sensors of the AR mobile device 12 not shown in Figure 1. In some embodiments, the positioning and orientation module 44 uses the aggregated sensor data to determine the position and orientation without relying on GPS, GNSS, WiFi positioning (e.g., trilateration), or any other positioning technique that requires a communication link between the AR mobile device 12 and other devices or systems. "Position" or "location" may refer to a specific set of coordinates in a 3D coordinate system (e.g., Cartesian or polar coordinates), and "orientation" may refer to a specific direction (e.g., a 360-degree horizontal / azimuth range, and altitude or elevation). In some embodiments, “orientation” may further refer to the tilt of the AR mobile device 12, regardless of the direction the device 12 is facing. Since only relative position is tracked (for example, in the sense of “dead reckoning”), the positioning and orientation module 44 determines the AR mobile device / user position relative to the location of the registered landmark.

[0043] In some embodiments, the positioning and orientation module 44 processes the fusion of data from at least the camera 24 and the IMU 26 to help overcome the shortcomings associated with either type of sensor when used alone. For example, the positioning and orientation module 44 may track the position and orientation of the AR mobile device 12 using Visual Inertial Measurement (VIO) technology developed by Qualcomm®. Such technology may help improve accuracy, reduce "drift" at the determined position, and / or have other advantages.

[0044] Since the position and orientation of the camera 24 relative to the AR mobile device 12 itself are known, the AR application 32 can determine the field of view of the AR mobile device 12 (which in some embodiments may also correspond to the user's view of the real world) for any given position and orientation determined by the positioning and orientation module 44. Based on the determined position and orientation, the node positions stored in the AR database 72 are mapped to a 3D map. Therefore, the AR application 32 can determine which mapped objects are in the field of view at any given time. In some embodiments, an object may be considered "in the field of view" of the camera if it is within the horizontal and vertical range / degree of the camera sensor (e.g., within a particular azimuth angle and vertical / altitude / elevation angle), regardless of the distance between the object and the AR mobile device capturing the image, and regardless of whether the object is incidentally obstructed by an obstacle in the environment. For example, an object directly and indirectly in front of camera 24 may be considered "in the field of view" of camera 24, even if the object is obscured by an enclosure, barrier, other object, etc. In other embodiments, an object is only considered "in the field of view" of the camera if the object is not obscured, i.e., while the camera can capture an image of the object.

[0045] To determine the location of mapped objects, the AR application 32 may periodically access node locations in the AR database via the network 16 and the web service interface 60. For example, the AR application 32 may periodically request the backend server 14 to provide node location data within a threshold distance of the AR mobile device 12 (and / or within the device 12's field of view, etc.), and the request may indicate the current location (and / or orientation) of the AR mobile device 12. Alternatively, the AR application 32 may request the backend server 14 to send all node locations (and, if possible, other information such as node descriptions) for a 3D map associated with a landmark used for registration, for example, during registration. In yet another embodiment, the backend server 14 may automatically send all relevant node locations when the AR mobile device 12 successfully registers with a landmark.

[0046] The AR application 32 may determine whether the augmentation should be provided to one, two or more of the mapped objects / nodes within the user's current field of view, or not to any of them. To make this determination, the node experience module 46 may apply one or more criteria. In some embodiments, for example, the node experience module 46 may determine that the augmentation should be provided to all mapped objects within the user's current field of view, regardless of the distance between the object and the user, and whether the object is obscured from the user's view by any obstacle. Alternatively, the node experience module 46 may determine that the augmentation should not be provided to mapped objects that are within the field of view but are beyond a threshold distance from the user (for example, determined from the current user position determined from the positioning and orientation module 44 and the corresponding node position stored in the AR database 72).

[0047] The node experience module 46 may also determine, or instead, that an augmentation should not be provided for mapped objects that are within the field of view of the AR mobile device 12 but are obscured from the field of view. Whether a mapped object is obscured may be determined in different ways depending on the embodiment. For example, a field in the AR database 72 may indicate whether a particular node corresponds to an object that is generally obscured. For example, in the case of a component enclosed within a cabinet, the corresponding node may have a field indicating the relationship between the component and the cabinet. In other embodiments, the AR application 32 and / or backend server 14 may perform a more complex analysis of node configuration, node type, and / or node size to determine whether a particular mapped object would likely be visible from the user's current viewpoint.

[0048] In some embodiments, to reduce the risk of sensory overload for the user, the augmentation will not extend to certain types of mapped objects unless the user takes a specific action (beyond simply moving and / or reorienting the AR mobile device 12). For example, the node experience module 46 cannot provide extensions to many of the relatively small components within the cabinet, each corresponding to a different mapped object, unless the user selects a cabinet icon or other graphic representation, and / or selects an option to show the contents of the cabinet.

[0049] Furthermore, in some embodiments, the node experience module 46 may determine, based on the user's role, whether an extension should be provided for a particular mapped object within the user's field of view, as determined by the role identification module 40. Thus, the node experience module 46 may query the web service interface 60, in particular, for nodes associated with both landmarks and selected or assigned roles. In a process control environment, for example, a power switch may be extended for a user with the "maintenance" role, but not for a user with the "operator" role.

[0050] For objects to be extended, the node experience module 46 may initially overlay node "markers," such as text (e.g., device identifiers, states, and / or descriptions) and / or icons or other graphics, onto an area of ​​the display 22 that appears to the user, as if they were located at (or near) the coordinates of the object in the user's real-world view. The markers may appear to be connected to the object mapped in the real-world view, for example, by lines rendered on the display 22. In other embodiments, the markers may be circles or other shapes rendered on the object, rectangular outlines roughly surrounding the object, or some other type of indicator.

[0051] When a user selects a particular node (for example, by converging a virtual reticle on a node marker or by issuing a voice command), the node experience module 46 may make one or more “node experiences” available to the user. In contrast to the more general “AR experience” of a well-equipped user navigating through a mapped environment, a “node experience” refers to a specific type of user interaction with an object / node. The node experience module 46 may provide the user with the option to select a particular node experience, for example, by overlaying a menu or other interactive display onto a real-world view and / or by being configured to recognize voice commands from the user. The user may select menu options in a similar manner to selecting a node marker (e.g., converging a virtual reticle on an option, a voice command, etc.) or in another appropriate manner.

[0052] Once a specific node experience is selected, the node experience module 46 may transmit the selection to the web service interface 60 via the network 16. In response, the experience provisioning unit 62 may retrieve data (e.g., text, graphics, etc.) associated with the selected node and selected experience from the AR database 72 (and / or other locations such as the server 18), and send the retrieved data back to the AR mobile device 12 via the web service interface 60, enabling the node experience module 46 to extend the user's view of the real world accordingly.

[0053] Node experiences can be relatively simple or complex. Node markers themselves can be considered, for example, the default “node experience.” As another example, node experience module 46 may display other text and / or tables (i.e., information associated with the corresponding object) in the user’s real-world view (via display 22), including simple graphics or icons of the object, surreal or pseudo-realistic 3D models of the object, images of the object, and animations containing the object (e.g., It is possible to overlay objects (such as rotating 3D models).

[0054] The node experience may also, or instead, include other types of interactivity. For example, the node experience module 46 may overlay links (e.g., URLs) to videos providing tutorials on objects and / or their behavior within the environment (e.g., within a specific process control routine), to work orders containing objects, or to remote experts. Alternatively, or in addition, the user may issue voice commands for some or all of these node experiences. In an implementation where content such as a video or work order is selected or invoked, the node experience module 46 may overlay the content onto the user's real-world view. If a remote expert or other person is selected or invoked, the node experience module 46 may enable the establishment of a communication link with the person over network interface 20 or another suitable network interface (e.g., via network 16), and in some embodiments, a still image or video of the person may be overlaid until communication is completed.

[0055] Other node experiences may enumerate and / or graphically depict certain types of relationships between selected objects / nodes and other objects / nodes. For example, in a single node experience, node experience module 46 may overlay a list of objects / nodes related to a selected object / node (e.g., parent and / or child objects / nodes) onto the user's real-world view. "Relationships" may be defined in any way desired, and may be set manually by the user when adding new nodes, or in another suitable way. For example, some relationships may refer to whether an object is a component of another object and / or whether it itself contains many components. A valve may, for example, correspond to a node that is the parent of a group of nodes corresponding to each component of the valve.

[0056] The node experience module 46 can determine relationships by querying the experience provisioning unit 62 via the web service interface 60, and the experience provisioning unit 62 retrieves the identifiers of the relevant nodes from the AR database 72 and provides the node experience module 46 with the metrics for those nodes. By default, the user can be spared from receiving an overwhelming amount of visual information at once by not displaying node markers and / or node experiences for all individual valve components (for example, unless a node or particular option is specifically selected by the user).

[0057] Several types of relationships can be graphically represented to the user. For example, the node experience module 46 may provide a graphical representation of the physical or logical connections between an object corresponding to a selected node and one or more other objects corresponding to other nodes. The node experience module 46 may determine connections by querying the experience provisioning unit 62 via the web service interface 60, which then retrieves identifiers of the connected nodes from the AR database 72 and provides the node experience module 46 with indicators of those nodes. The node experience module 46 may then generate a display that draws lines connecting to the appropriate objects and overlays these lines onto the user's real-world view. An example of such a node experience is provided in Figure 4H, which is described below.

[0058] The connections and / or other relationships shown may also change based on the user's selected or assigned role. In a process control environment, for example, an "operator" might be shown a line from a sensor / transmitter device to a tank, indicating that the sensor / transmitter measures the pressure in the tank. Conversely, an "engineer" might instead (or in addition) show a line from the sensor / transmitter to another device that receives the sensor transmission. It is also possible that the "maintenance worker" may instead (or in addition) be shown the wire from the sensor / transmitter to the device's power supply (for example, so that the maintenance worker can easily find where to cut off the power before performing maintenance, repair, or replacement of the device).

[0059] In some embodiments, the experience provisioning unit 62 utilizes data and / or functions from other servers 18(or more) and / or applications 19(or more) to provide one or more node experiences, and / or one or more applications 19(or more) may be invoked in response to data sent from the experience provisioning unit 62. Several examples of such embodiments are described below in a process control environment with reference to Figure 2.

[0060] As described above, the node experience module 46 may determine whether to extend the user's real-world view of a particular mapped object based on the user's selected or assigned role (e.g., operator, maintainer, etc.). In addition, or alternatively, the type of node experience, and / or the content or interactivity provided by a particular experience, may vary based on the user's role. For example, a user with the role of "maintainer" and who has a pump in their real-world view may be presented with a node experience showing scheduled maintenance reminders for the pump, while a user with the role of "operator" or "engineer" may instead be presented with information about the logical connections of the pump within process control routines. If scheduled maintenance is delayed, the maintainer may be shown an alert, while the operator or engineer may only be shown alerts in other scenarios (e.g., if the pump fails).

[0061] The node creation module 48 supports adding new nodes in the 3D map corresponding to landmarks used to register the AR mobile device 12. In some embodiments, any user can add a new node to the 3D map. In other embodiments, only specific user roles and / or only specific AR devices may be used to add new nodes. To add a new node, the user may choose the option to add the object as a new node by orienting the field of view of the AR mobile device 12 (e.g., a reticle located in the center of the field of view) towards the real-world object to be mapped.

[0062] This can be achieved in different ways depending on the embodiment. For example, the user may select an overlaid menu item (e.g., a virtual "Add Node" button) or issue a voice command, in which case the node creation module 48 may overlay a reticle onto the user's real-world view. While pointing the reticle at an object, the user may activate another control (e.g., a virtual "Confirm" button) or issue a second voice command. In response, the node creation module 48 may determine the object's location and prompt the user to enter a node name and / or description. The node creation module 48 may then transmit the location, entered or spoken name, etc., to the web service interface 60 via the network 16, and the 3D model generation unit 64 may transmit at least the location and node identifier (e.g., the node name entered or spoken by the user) to the AR database 72.

[0063] To determine the position of the mapped object, the node creation module 48 may use the current position of the AR mobile device 12 (as determined by the positioning and orientation module 44) and offset its position based on both the distance between the AR mobile device 12 and the object, and the orientation of the AR mobile device 12. This can be achieved in a similar manner to determining the initial 3D position of the AR mobile device 12 when registering it as a landmark, as described above. For example, the node creation module Module 48 may process the image of the object captured by the camera 24 using depth sensing technology to determine the distance between the AR mobile device 12 and the object. The positioning and orientation module 44 may then offset the position of the object from the position of the AR mobile device 12 based on both the determined distance and orientation of the AR mobile device 12 (for example, the distance and direction the AR mobile device 12's field of view was facing when the image of the object was captured).

[0064] In some embodiments, the node creation module 48 may also be used to update / modify existing nodes. For example, a menu provided by the node experience module 46 may include an option to update node information. If selected, the node creation module 48 may overlay one or more menus or other user interfaces on the user's real-world view to allow the user to change information about a node (e.g., node name, node description, etc.). Any changes may be sent to the web service interface 60 via the network 16, and the 3D model generation unit 64 may update the AR database 72 accordingly.

[0065] In some embodiments, the AR application 32 configures the AR mobile device 12 to enhance the user's virtual vision beyond the extensions described above. For example, for certain types of nodes, the vision enhancement module 50 may determine when a mapped object is located within an area that is captured by the user's current field of view (e.g., as described above for the node experience module 46) but obscured by one or more objects, and provide the user with a virtual "X-ray vision." The virtual X-ray vision may be provided only if one or more other criteria are established (e.g., if the vision enhancement module 50 determines that the hidden object is within a threshold distance of the AR mobile device 12), or it may be provided regardless of any other criteria.

[0066] In one such embodiment, the AR database 72 includes, for each of at least some of the node inputs, a flag or other visibility indicator indicating that the node may be hidden from any user's view. This visibility indicator may be automatically generated based on known relationships of objects (e.g., if the 3D model generation unit 64 accesses a control system to know that the object corresponding to the node is located within a cabinet), or it may be set manually by the user (e.g., via a user interface that the node creation module 48 overlays onto the user's real-world view when adding a node). If a particular mapped object is within the user's field of view, the visual enhancement module 50 may query the visual enhancement unit 66 on the backend server 14 via the network 16 and the web service interface 60, which may then access the AR database 72 to retrieve the relevant visibility indicator. The visual enhancement unit 66 may then use the web service interface 60 to send the visibility indicator, or other data indicating whether the object is obscured, to the visual enhancement module 50.

[0067] Alternatively, the visual enhancement module 50 and / or the visual enhancement unit 66 may perform visibility determination in other ways. For example, a particular mapped object or set of objects may be located within a cabinet marked with a model number, QR code, or other visual indicator. The visual enhancement module 50 may process images of the visual indicators captured by the camera 24 to determine the cabinet identifier (e.g., node identifier) ​​and transmit that identifier to the visual enhancement unit 66 via the web service interface 60. The visual enhancement unit 66 then uses the identifier to determine which mapped object (e.g., field device, I / O card, etc.) is located within the cabinet. The system may determine if an object is inside a box and send back data indicating that the object inside the cabinet is unclear.

[0068] To provide the effect of virtual X-ray vision on a given object, the visual enhancement module 50 may search for a surreal or pseudo-realistic 2D or 3D model of the object, or a digital image or video of the object, and overlay the model, image, or video on (or near) the object in the user's field of view. Either by default or in response to user selections such as 2D or 3D models or voice commands, the node experience module 46 may also provide visual menu options or voice command recognition, as described above, allowing the user to select various experiences with the node. Thus, the user may interact with the hidden object in a way that appears and "feels" very similar to interacting with a real-world object while the real-world object is directly in their field of view.

[0069] The visual enhancement module 50 may also, or instead, enhance the user's virtual vision in other ways. For example, if an object is in a restricted-access and / or hazardous area (e.g., at a very high position, a barricaded location, a high-voltage or toxic area), and / or far from the user, the visual enhancement module 50 may allow the user to search for an “avatar” that represents the object in their field of view. The avatar may be the same as, or in some respects different from, the 2D or 3D models, images, videos, etc., described above for the example of X-ray vision. In fact, in some embodiments, virtual X-ray vision is just one of several use cases for invoking avatars of real-world objects.

[0070] Avatars may be invoked in different ways depending on the embodiment and / or scenario. In some embodiments, the visual enhancement module 50 first superimposes the visual indicators onto the user's real-world view if certain criteria are met, such that the visual indicators provide some indices of the location of the corresponding object relative to the user's real-world view. For example, the visual indicators may be presented if (1) the AR mobile device 12 is within some threshold distance of the object, and (2) the node is flagged as an object with restricted access. The AR database 72 may store data indicating such a state (for example, based on manual user input, such as when a node was added by the user), or the state may be inferred (for example, if the visual enhancement unit 66 determines that the object is in an area that the backend server 14 has flagged as a “dangerous” area, or if the visual enhancement module 50 or the visual enhancement unit 66 determines that the object's location is at least a threshold distance from the AR mobile device 12 in the z direction and therefore at a very different elevation angle). The visual indicators may include arrows pointing to the object, lines leading to the object, or some other indices of location. An example of a visual indicator is described below in relation to Figure 7A. If the user selects an indicator or takes another appropriate action (for example, in some embodiments, if the user changes their field of view to include an object), the visual enhancement module 50 may overlay the avatar onto the user's real-world view.

[0071] In other embodiments, the indicator of the object's location includes the avatar itself (e.g., having an arrow / pointer to the object's location on or near the avatar). In yet other embodiments and / or scenarios, the visual enhancement module 50 overlays the avatar onto the user's field of view without presenting a visual indicator of the object's location. For example, the visual enhancement module 50 responds to the user requesting or searching for a corresponding object (e.g., by issuing a voice command or manually entering data). An avatar may be presented to the user in response to actions such as the user entering a "Ta" or selecting a node marker for an object that is relatively far away (e.g., beyond a certain threshold distance). In one such embodiment, the avatar does not appear immediately at full size. For example, the visual enhancement module 50 may create a visual effect in which the avatar appears to the user as if it were moving from the object's location (e.g., away from the user's front) to a position just in front of the user. The avatar may be enlarged, for example, to simulate a drawing effect that is closer to the user.

[0072] The node experience module 46 may automatically provide a specific node experience to an object for the user (in addition to showing the object's avatar), and / or allow the user to select one or more node experiences after the avatar is initially presented (for example, by issuing a voice command or selecting a menu option). For example, one or more of the node experiences described above may be provided. An example of an experience that may be associated with an avatar is described below in relation to Figure 7B.

[0073] In some embodiments, the AR application 32 configures the AR mobile device 12 to generate alerts based on the user's current location and / or the current situation in a specific area (e.g., a specific area of ​​a process plant is associated with an alert). The alert generation unit 68 may periodically or continuously determine whether the current location of the AR mobile device 12 is in an area that should receive an alert or warning, for example, by comparing the location reported by the positioning and orientation module 44 (e.g., via the web service interface 60) with the boundaries of one or more geographically isolated areas. If the user / device is in an area associated with an alert, the alert generation unit 68 may send an indicator of the alert to the location-based alert module 52 via the web service interface 60, which may overlay the indicator of the alert (e.g., text and / or graphics) on the user's real-world view, trigger some other visual effects (e.g., a flashing red light or hue encompassing the entire real-world view), and / or provide the user with an audio alert (not shown in Figure 1, but via the speaker of the AR mobile device 12). Alternatively or additionally, the alert generation unit 68 may send alert indicators to specific mapped objects based on the proximity of the AR mobile device 12 (for example, if an object is malfunctioning and requires maintenance or should be avoided). The alert generation unit 68 may also condition the delivery of alerts regarding a user's selected or assigned role.

[0074] In some embodiments, the backend server 14 may monitor the 3D position of the AR mobile device (and therefore the user) within the mapped environment. For this purpose, the user monitoring unit 70 may record the current and / or past positions of users who have registered their AR mobile device as landmarks, based on data received from the positioning and orientation module 44 via the web service interface 60.

[0075] User location monitoring can be used for a variety of purposes, depending on the embodiment and / or necessity. For example, the user monitoring unit 70 may plot the user's location on a display or terminal (not shown in Figure 1) coupled to the backend server 14, and the operator of the display or terminal may describe the location of an employee in an emergency, or simply monitor the user under more general conditions to determine whether the employee requires medical assistance. The employee's health status may also be monitored using other data, such as biometric data collected by one or more sensors of the AR mobile device 12 (e.g., to detect pulse, blood pressure, temperature, etc.). As a further example, employee / user location may be added to the employee's training log and used by an administrator or HR person to track protocol compliance. It can be used to monitor conditions, etc. In other embodiments and / or scenarios, the 3D location of specific non-human mobile assets within the mapped environment may be tracked. For example, a unit of the backend server 14 may monitor the 3D location of mobile reactors, carts, field service trucks, and / or other objects for various purposes.

[0076] Figure 1 and the above description represent only a few potential embodiments, and it should be understood that others are also possible. For example, the AR platform may not use a cloud-based architecture or web-based services. As another example, some or all of the functionality of units 62 to 68 of the backend server 14 and / or the AR database 72 itself may be partially or entirely incorporated into the AR mobile device 12. As yet another example, node markers and / or node experiences may be provided in a non-AR context, e.g., a virtual reality (VR) context, or in relation to a non-AR, non-VR graphical user interface (GUI).

[0077] Furthermore, the AR platform may offer other features not mentioned above. For example, users of an AR mobile device may add field notes associated with various nodes / objects in the AR database 72 (e.g., using voice commands) and / or start new work orders associated with various nodes / objects, etc.

[0078] Exemplary process control environment Figure 2 is a block diagram of an exemplary process control environment 100 in which the AR system 10 of Figure 1 may be used. The process control environment 100 (also referred to herein synonymously as process control system 100 or process plant 100) includes one or more process controllers that receive signals indicating process measurements taken by field devices, process this information to implement control routines, and generate control signals transmitted to other field devices via wired and / or wireless process control communication links or networks in order to control the operation of processes within the plant 100. Typically, at least one field device controls the operation of the process by performing a physical function (e.g., opening and closing a valve, moving a substance on a conveyor, raising or lowering a temperature, acquiring a measurement, sensing a state, etc.). Some types of field devices communicate with controllers using I / O devices. The process controllers, field devices, and I / O devices may be wired or wireless, and any number and combination of wired and wireless process controllers, field devices, and I / O devices may be included in the process plant environment or system 100.

[0079] For example, Figure 2 illustrates a process controller 111 that is communicatively connected to wired field devices 115-122 via input / output (I / O) cards 126 and 128. The process controller 111 includes a processor 130, memory 132, and one or more process control routines 138, which will be discussed in more detail below. The controller 111 is also communicatively connected to wireless field devices 140-146 via a process control communication network or backbone 110 and a wireless gateway 135. The backbone 110 may include one or more wired and / or wireless communication links, which may be implemented using any suitable communication protocol, such as the Ethernet protocol. In some configurations (not shown in Figure 2), the controller 111 supports one or more communication protocols, such as the IEEE 802.11 standard wireless local area network protocol, mobile communication protocols (e.g., WiMAX, LTE, etc.), Bluetooth®, HART®, WirelessHART®, Profibus, FOUNDATION® fieldbus, etc. The wireless gateway 135 may be communicatively connected to the backbone 110 using one or more other communication networks, such as using a number of other wired or wireless communication links.

[0080] Controller 111 (for example, a DeltaV® controller sold by Emerson Process Management) may operate to implement a batch or continuous process using at least some of the field devices 115-122 and 140-146. In one embodiment, in addition to being communicatively connected to the backbone 110, Controller 111 may also be communicatively connected to at least some of the field devices 115-122 and 140-146 using any desired hardware and software, which may be associated with any suitable smart communication protocol such as standard 4-20mA devices, I / O cards 126, 128, and / or FOUNDATION® fieldbus protocol, HART® protocol, WirelessHART® protocol. In Figure 2, Controller 111, field devices 115-122, and I / O cards 126, 128 are wired devices, and field devices 140-146 are wireless field devices. Naturally, the wired field devices 115-122 and the wireless field devices 140-146 can comply with any other desired standard(s) or protocol, such as any preferred wired or wireless protocol, including any preferred standard or protocol to be developed in the future.

[0081] The processor 130 of the process controller 111 implements or supervises one or more process control routines or modules 138 that can be stored in memory 132. For this purpose, the processor 130 is configured to communicate with field devices 115-122 and 140-146, as well as with other nodes that are communicably connected to the controller 111. Note that any control routine or module described herein may, if so desired, be implemented or executed in part by a different controller or other device. Similarly, the control modules 138 implemented in the process control system 100 may be in any form, including software, firmware, hardware, etc. The control routines may be implemented in any desired software format, such as object-oriented programming, ladder logic, sequential function charts, functional block diagrams, or any other software programming language or design paradigm. The memory 132 in which some or all of the control modules 138 can be stored may be any preferred type of memory or memory(s), such as random access memory (RAM) and / or read-only memory (ROM). Furthermore, the control module 138 may be hardcoded into, for example, one or more EPROMs, EEPROMs, application-specific integrated circuits (ASICs), or any other hardware or firmware elements. Thus, the controller 111 may be configured in any desired manner to implement the control strategy or control routines / modules.

[0082] The controller 111 implements the control strategy using what are generally called functional blocks, each functional block being an object or other part (e.g., a subroutine) of the overall control routine, and working together with other functional blocks (via communication called links) to implement the process control loop within the process control system 100. Control system functional blocks typically perform one of the following: input functions, such as those associated with transmitters, sensors, or other process parameter measuring devices; control functions, such as those associated with control routines that perform control such as PID or fuzzy logic; or output functions that control the operation of several devices, such as valves or conveyor motors, thus performing several physical functions within the process control system 100. Naturally, hybrid and Other types of functional blocks exist. These functional blocks are typically used in or associated with standard 4-20mA devices and certain types of smart field devices (e.g., HART® devices), and may be stored and executed by the controller 111, or, as may be the case with FOUNDATION® fieldbus devices, may be stored and implemented by the field device itself. One or more control modules 138 within the controller 111 may implement one or more control loops that are performed by executing one or more of the functional blocks.

[0083] The wired field devices 115-122 may be any type or combination of devices such as sensors, valves, conveyor motors, transmitters, and positioners, while the I / O cards 126 and 128 may be any type of I / O device conforming to a suitable communication or controller protocol. For example, the field devices 115-118 may be standard 4-20mA devices or HART® devices that communicate with the I / O card 126 via an analog line (or a combination of analog and digital lines), while the field devices 119-122 may be smart devices such as FOUNDATION® fieldbus field devices that communicate with the I / O card 128 via a digital bus using the FOUNDATION® fieldbus communication protocol. However, in some embodiments, at least some of the wired field devices 115-122 and / or at least one of the I / O cards 126, 128 additionally or alternatively communicate with the controller 111 using the backbone 110 and a preferred control system protocol (e.g., Profibus, DeviceNet, Foundation fieldbus, ControlNet, Modbus, HART, etc.).

[0084] In Figure 2, wireless field devices 140-146 communicate via the wireless process control communication network 170 using a wireless protocol such as the WirelessHART® protocol. Such wireless field devices 140-146 may communicate directly with one or more other devices or nodes in the wireless network 170, which are also configured to communicate wirelessly. To communicate with other nodes that are not configured to communicate wirelessly, wireless field devices 140-146 may utilize a wireless gateway 135 connected to the backbone 110 or another process control communication network. The wireless gateway 135 provides access from the backbone 110 to various wireless devices 140-158 in the wireless communication network 170. Specifically, the wireless gateway 135 provides communication connectivity between wireless devices 140-158, wired devices 115-122, and / or other nodes or devices in the process control plant 100.

[0085] Similar to the wired field devices 115-122, the wireless field devices 140-146 of the wireless network 170 perform physical control functions within the process plant 100, such as opening and closing valves and obtaining measurements of process parameters. However, the wireless field devices 140-146 are configured to communicate using the wireless protocol of the network 170. As such, the wireless field devices 140-146, the wireless gateway 135, and the other wireless nodes 152-158 of the wireless network 170 can be producers and consumers of wireless communication packets.

[0086] In some configurations of the process plant 100, the wireless network 170 includes non-wireless devices. For example, in Figure 2, field device 148 may be an old 4-20mA device, and field device 150 may be a wired HART® device. To communicate within the network 170, field devices 148 and 150 Each wireless device is connected to the wireless communication network 170 via one of the wireless adapters 152A or 152B. The wireless adapters 152A and 152B support wireless protocols such as WirelessHART and may also support one or more other communication protocols such as Foundation® Fieldbus, Profibus, and DeviceNet. Furthermore, in some configurations, the wireless network 170 includes one or more network access points 155A, 155B, which may be separate physical devices communicating via wired connections with the wireless gateway 135, or may be integrated within the wireless gateway 135. The wireless network 170 may also include one or more routers 158 that forward packets from among the wireless devices within the wireless communication network 170. The wireless devices 140-146 and 152-158 may communicate with each other and with the wireless gateway 135 via the wireless links 160 and / or the backbone 110 of the wireless communication network 170.

[0087] In Figure 2, the process control system 100 includes one or more operator workstations 171 that are communicatively connected to the backbone 110. Through the operator workstations 171, human operators can perform any necessary diagnostics, adjustments, maintenance, and / or other actions, in addition to monitoring the runtime operation of the process plant 100. At least some of the operator workstations 171 may be located in various protected areas within or near the plant 100, such as the backend environment of the plant 100, and in some cases, at least some of the operator workstations 171 may be located in remote locations (but still communicatively connected to the plant 100). The operator workstations 171 may be wired or wireless computing devices.

[0088] An exemplary process control system 100, further illustrated in Figure 2, includes one or more configuration applications 172A and one or more configuration databases 172B, each of which is also connected to the backbone 110 in a communicative manner. Various instances of the configuration application 172A may run one or more computing devices (not shown in Figure 2) to enable the user to create or modify process control modules and download these modules to the process controller 111 and / or other process controllers via the backbone 110, and to enable the user to create or modify operator interfaces by allowing operators to view data and change data settings within process control routines. The configuration database 172B stores the configured modules and / or operator interfaces. Generally, the configuration applications 172A and configuration databases 172B may be centralized and have a single logical appearance to the process control system 100 (even though multiple instances of the configuration applications 172A may run simultaneously within the process control system 100), while the configuration databases 172B may be stored in a single physical data storage device or across multiple data storage devices. The configuration applications 172A, configuration databases 172B, and their user interfaces (not shown in Figure 2) together form a configuration or development system 172 for creating / configuring control and / or display modules. Typically, but not necessarily, the user interface of the configuration system 172 differs from the operator workstation 171 in that it comprises a user interface for the configuration system 172 used by configuration and development engineers regardless of whether the plant 100 is operating in real time, and an operator workstation 171 used by operators during the real-time (or "runtime") operation of the process plant 100.

[0089] The exemplary process control system 100 also includes one or more data historian applications. This also includes one or more instances 173A and one or more data historian databases 173B, each of which is communicably connected to the backbone 110. The data historian application 173A operates to collect some or all of the data provided through the backbone 110 and store that data in the data historian databases 173B for long-term storage. Similar to the configuration applications 172A and configuration databases 172B, the data historian applications 173A and data historian databases 173B may be centralized and have a single logical appearance to the process control system 100 (even though multiple instances of the data historian application 173A may run simultaneously within the process control system 100), while the data historian databases 173B may be stored in a single physical data storage device or across multiple data storage devices. The data historian application 173A, the data historian database(s) 173B, and their user interfaces (not shown in Figure 2) together form the data historian system 173.

[0090] In some configurations, the process control system 100 supports other wireless protocols such as the IEEE 802.11 standard Wireless Local Area Network Protocol, WiMAX (Worldwide Interoperability for Microwave Access), LTE (Long Term Evolution), or other ITU-R (International Telecommunication Union) protocols. The system includes one or more other wireless access points 174 that communicate with other devices using mobile communication protocols such as Radiocommunication Sector (RCS) compatible protocols, short-wavelength wireless communication such as Near Field Communication (NFC) or Bluetooth, and / or other wireless communication protocols. Typically, such wireless access points 174 enable handheld or other portable computing devices (e.g., user interface devices 175) to communicate by their respective wireless process control communication networks, which, unlike the wireless network 170, support different wireless protocols other than the wireless network 170. For example, the wireless or portable user interface device 175 may be a mobile workstation or diagnostic test equipment used by an operator within the process plant 100 (e.g., one instance of an operator workstation 171). In some scenarios, in addition to portable computing devices, one or more process control devices (e.g., a controller 111, field devices 115-122, wireless devices 135, 140-158, etc.) also communicate using the wireless protocols supported by the wireless access points 174.

[0091] Figure 2 illustrates only a single process controller 111, a certain number of field devices 115-122 and 140-146, a wireless gateway 35, a wireless adapter 152, an access point 155, a router 1158, and a wireless process control communication network 170 included in an exemplary process plant 100. It should be noted that this is only an exemplary and non-limiting embodiment. For example, any number of controllers 111 may be included in the process control plant or system 100, and any of the controllers 111 may communicate with any number of wired or wireless devices and networks 115-122, 140-146, 135, 152, 155, 158, and 170 to control processes within the plant 100.

[0092] Returning to Figure 1, the objects mapped using the AR mobile device (for example, AR mobile device 12) are the controller 111, devices 115-122 and 140-158, I / O cards 126 and 128, wireless gateway 135, and This includes some or all of the other devices and / or parts of the equipment within Plant 100, and as a result, the 3D locations within Plant 100 (and optionally their relationships to each other, their avatars, etc.) are stored in the AR database 72. In some embodiments, the backend server 14 is communicably coupled to the backbone 110, and other servers 18 may include, or be coupled to, operator workstations 171, configuration systems 172, data historian systems 173, and / or other computing devices or systems in the process control environment 100.

[0093] The applications 19 executed by the server(s) 18 may include one or more DeltaV® applications, diagnostic and maintenance applications, and / or other applications or software-based systems. Thus, the applications 19 may support a node experience that displays the runtime status and / or measurement data of field devices, provides runtime control of field devices, and displays diagnostic / maintenance information of machines. As an example, HART® and / or fieldbus information related to various devices may be presented to the user of the AR mobile device 12. The experience provisioning unit 62 may launch and / or interface with the appropriate application(s) 19 to collect the necessary display data, which is then forwarded to the node experience module 46. The experience provisioning unit 62 may also, or alternatively, forward any user input received from the node experience module 46 to the controller 111 to control the appropriate field devices, etc., accordingly.

[0094] Exemplary 3D map From the explanation of Figure 1 above, it should be understood that the 3D "map" of the AR system 10 does not need to be a complete model of the environment, but may only contain the 3D locations of nodes within that environment. A very simplified exemplary 3D map 200 containing only four objects / nodes is shown in Figure 3. In Figure 3, the z-axis corresponds to the elevation angle (e.g., ground level). As mentioned above, the landmark location 202 may provide a reference point for all other locations in the environment (e.g., nodes and user / AR device locations).

[0095] In the example map 200, the first position 204 of the first object / node is slightly above (in the elevation direction) the landmark position 202 at offset {x1, y1, z1}, the second and third positions 206 and 208 of the second and third objects / nodes are still a little higher at offsets {x2, y2, z2} and {x3, y3, z3}, respectively, and the third position 210 of the third object / node is still higher at offset {x4, y4, z4}. Naturally, in some scenarios, the offsets in the x, y and / or z directions can be negative for a particular object / node.

[0096] Within map 200, position 220 at {x5, y5, z5} corresponds to the current position of an AR mobile device, such as the AR mobile device 12 in Figure 1. Figure 3 also depicts the field of view 222 of the AR mobile device, represented by a dashed line, corresponding to a specific orientation of the device. The field of view 222 can be determined by the positioning and orientation module 44 in Figure 1, for example, using sensor data from the IMU 26 (and possibly image data from the camera 26 using VIO technology), as well as the camera 26 with a known azimuth range / angle. Although not shown in Figure 3, the positioning and orientation module 44 can also know the range of the field of view 222 in the z direction, for example, so that the field of view 222 has a conical shape rather than assuming infinite coverage in the z direction.

[0097] In this example scenario, the objects at positions 206 and 208 are within the device / user's current field of view 222, while the objects at positions 204 and 210 are not present. The experience provisioning unit 62 in Figure 1 may make a determination based, for example, on the node location (stored in the AR database 72), the AR mobile device's position 220 and orientation (determined by the positioning and orientation module 44 and transmitted via the web service interface 60), and known or estimated characteristics of the camera 24's field of view. Based on the determination of which objects / nodes are within the field of view 222, the experience provisioning unit 62 may send data to the AR mobile device 12 indicating that node markers (and / or specific node experiences) should be presented only for the objects at positions 206 and 208. The node experience module 46 may process this data and, in response, overlay the node markers corresponding to positions 206 and 208 and / or other information on or near the point on the display 22.

[0098] Exemplary AR user interface A set of exemplary user interfaces that can be superimposed on an AR display (e.g., the display 22 of the AR mobile device 12 in Figure 1) is described herein in relation to Figures 4A to 4H. For ease of explanation, the AR system 10 in Figure 1 is also referred to, although the user interfaces shown may be used in other systems instead. For user interfaces in which user selection may be made, the selection may be made by the user directly viewing a particular menu option (e.g., if the user interface has a fixed position relative to the environment rather than a fixed position relative to the center of the display 22). In other embodiments, the selection may be made by the user issuing an appropriate voice command, entering data on a keyboard or touchscreen, or by another suitable method.

[0099] Referring first to Figure 4A, the user interface 200 allows the user to select a specific "role," as described above in relation to Figure 1. The user interface 200 may be displayed to the user when the AR application 32 is launched, or at another appropriate time. The AR application 32 may a priori know of the available roles, or be notified about the available roles, by querying the backend server 14 via the web service interface 60. Once a role is selected, the role identification module 40 may send data indicating the role to the backend server 14 via the web service interface 60. The experience provisioning unit 62, the alert generation unit 68, and / or other components of the backend server 14 may then use the selected role to determine the appropriate nodes to extend, the appropriate types of experiences, and / or content for specific nodes, as described above.

[0100] Figure 4B depicts a user interface 220 that can function as a node marker for several or all objects (subject to proximity or other criteria such as user role) within the user's current field of view. The exemplary user interface 220 includes an icon (a circled "X") and descriptor ("StatusWord") indicating the status of an object / node, a node name (e.g., a device identifier in a process control plant, such as "ESD-3"), a short description of the node (e.g., "Emergency Stop Switch"), and a vertical line from that information to a filled circle at the bottom of the user interface 220. The filled circle (or other indicator) may be positioned at the {x, y, z} location of the corresponding object in the user's real-world view, for example, at the real-world {x, y, z} location where the user (or another user) adds the node to the 3D map. The text and lines may be rotated as needed, or may always be aligned to the z axis, for example, to remain aligned with the vertical axis of the AR mobile device 12 and the user.

[0101] State icons and descriptors may indicate whether some action should be taken on an object and / or whether there is any danger to the object. For example, the circled "X" shown in Figure 4B may correspond to the "danger" state, the triangle "!" may correspond to the "warning" state, and the diamond-shaped "i" may correspond to the "caution" state; no icon may correspond to the "normal" state. Figure 4B shows the range of possible state icons, but it is understood that the user interface 220 will include only the circled "X" icon for this particular embodiment and / or scenario. The states drawn on a particular object / node and the meaning of those states may vary depending on the user's role.

[0102] The node markers of the user interface 220 (i.e., the text, lines, and filled circles in Figure 4B) may be enlarged in size depending on the distance from the AR mobile device 12, so that closer objects have larger node markers and farther objects have smaller node markers. In some embodiments, for example, the user interface 220 may include more information through additional text, additional graphics, and / or color coding (e.g., red text and lines for a "dangerous" state, green text and lines for a "normal" state, etc.).

[0103] Figure 4C depicts User Interface 240, which may appear after a user role is selected in User Interface 200, or at another appropriate time. The "+" icon in User Interface 240 may be selected by the user to add a new node to the 3D map, as described above in relation to Figure 1. The helmet icon may be selected by the user to select a different role, and a third icon may be selected by the user to manage specific settings, as described later. Rectangular boxes within User Interface 240 may display system messages to the user. For example, a message box may be used to display alerts generated by the alert generation unit 68 and / or the location-based alert module 52.

[0104] Figure 4D depicts the user interface 260 that may appear when the user selects a settings icon within the user interface 240. As shown in Figure 4D, the user interface 260 allows the user to change the name or identifier of the AR mobile device 12, the landmark identifier, and the name or IP address of the backend server 14. Some or all of these values ​​may be used by the AR mobile device 12, for example, when making a web service call to the web service interface 60.

[0105] Figure 4E shows a simple user interface 280 that may appear when the user selects the "+" (add node) icon in user interface 240. In this example, user interface 280 is simply a reticle (e.g., in the center of the user's display 22) that can be centered at any location in the real-world environment by the user simply looking around (e.g., if the AR mobile device 12 is an AR helmet or goggles, or smart glasses) or otherwise aiming the camera 24 of the AR mobile device 12 (e.g., if it is a tablet or smartphone). The node creation module 48 of the AR application 32 may require the user to confirm when the reticle is properly positioned (e.g., by issuing a voice command), or may assume proper positioning, such as when the field of view of the camera 24 does not substantially change over some threshold period.

[0106] Figure 4F shows the state after the user has positioned the reticle of the user interface 280. After the positioning and orientation module 44 determines the position and / or orientation information necessary to place a new object / node (e.g., the position of the AR mobile device 12, the orientation of the device 12, and the distance from the device 12 to the object), it draws a possible user interface 300. An exemplary user interface 300 includes a text box for the user to enter a node name and / or node description. Alternatively, some or all of this information may be provided by voice commands (e.g., if the user selects the microphone icon). In other embodiments, the node name and / or description may be determined by processing an image of part of the object using optical character recognition (OCR) technology. For example, the user may point the camera 24 towards a tag or label attached to the object that contains a serial number, model number, or other identifier. The identifier is then sent to the backend server 14 via the web service interface 60 and cross-referenced with information stored in the backend server 14, or stored in another system or database (e.g., another server 18) to determine the node name and / or description.

[0107] Figure 4G illustrates a user interface 320 that may appear for a particular object / node when the user selects that object / node (for example, by selecting the node marker in user interface 220). User interface 320 may be generated by the node experience module 46 based on data indicating the available experiences for the node provided by the experience provisioning unit 62 of the backend server 14. An exemplary user interface 320 includes the node name and description shown on the node marker in user interface 220, as well as two rectangular areas. The filled circle at the bottom of user interface 320 may remain in the same position on the object (in the real-world view) that was in user interface 220. The first “Working” area, indicated by a blank rectangle in Figure 4G, may display additional information about the selected object / node, such as the object's state information, detailed alert information for the object, etc. (for example, depending on the type of node experience being provided and / or the user's role).

[0108] The user may select the desired type of node experience from a second area and enumerate what is available for the node (and possibly the user's role). As seen in Figure 4G, in this example, the available node experiences include “Additional Information” (e.g., to provide additional types of text-based information in the work area), “Connections” (e.g., to indicate physical or logical connections to other objects / nodes, as will be discussed later in relation to Figure 4H), “Information Links” (e.g., to display information from other web pages at a specific URL in the work area, such as “Help” documents, instruction videos, or specific URLs), “Child Objects” (e.g., to show relationships with other objects / nodes related as “children” of the selected object / node in the work area), “Process Values” (e.g., to show control process values ​​obtained from one of the other servers 18 and applications 19 in the work area), and “Work Orders” (e.g., to show the text of work orders associated within the selected object / node in the work area).

[0109] Figure 4H depicts User Interface 340, which may appear when the user selects the "Connect" icon within User Interface 320. In User Interface 340, the Node Experience Module 46 renders a line between the {x, y, z} position of the currently selected object / node and the {x, y, z} position of the object / node to which the currently selected object / node is physically and / or logically connected (for example, as described above in relation to Figure 1). The filled circle at the bottom of User Interface 340 may be at the former position, while User Interface 34 The filled circle above 0 may be in the latter position. Naturally, the relative position of the filled circles may change depending on the scenario (for example, the circle of a selected object / node may appear higher in the real-world view than the circle of a connected object / node).

[0110] In the embodiment of Figure 4H, the node experience module 46 renders the connection line in three segments: a first segment showing distance and direction along the y-axis (starting at the currently selected object / node), a second segment showing distance and direction along the x-axis (starting at the end of the first segment), and a third segment showing distance and direction along the z-axis and ending at the connected object / node (starting at the end of the second segment). The line segments may be color-coded, encoded by a line format (dashed, solid, etc.), and / or encoded in other ways to provide additional information (e.g., a red line indicating a communication connection that is not currently working). In other embodiments, connections may be drawn in different ways, such as a single line segment between objects / nodes (e.g., the line widens and narrows to depict the distance from the user). In various different embodiments, connections to multiple different objects / nodes may be shown in the user interface 340, or the user may need to select each individual connection for which visualization is desired.

[0111] An example of an AR database Figures 5A and 5B collectively illustrate the database structure 400 that may be available to the AR database 72 in Figure 1 (for example, when used in the process control environment 100 in Figure 2). In the database structure 400, each box represents a table that may be stored in the AR database 72, and the top row of each box is the table name (e.g., "Node", "Node State", etc.). The tables are linked using primary keys, each key is shown in Figures 5A and 5B with a key icon and the letters "PK".

[0112] Database structure 400 enables efficient cross-referencing of data items, allowing user and system updates without breaking existing links or requiring extensive updates. For example, using a "Node ID" primary key allows users to change the "Node Name" for existing nodes without having to update other related tables. The "OPCDA Retrieval" table in Figure 5B may correspond to the "Process Value" node experience described above in relation to Figure 4G.

[0113] Exemplary emphasis on virtual vision Figures 6 and 7 relate to embodiments in which an AR platform (e.g., provided by the AR system 10 in Figure 1) provides a function that enhances the user's “virtual vision.” Figure 6 depicts a “virtual X-ray vision” function (described above in relation to Figure 1) used to “see” inside a closed cabinet within a process control environment (e.g., within the process control environment 100 in Figure 2), while Figures 7A and 7B depict a specific scenario in which an “avatar” (also described above in relation to Figure 1) is provided to a sensor / transmitter device in a restricted access area (e.g., hard to reach or dangerous) of the process control environment and / or away from the user’s current location.

[0114] First, referring to the example in Figure 6, a closed (and possibly locked) cabinet 500 may contain several internal components (not shown in Figure 6), such as I / O cards and circuit breakers. When any of the criteria described in relation to Figure 1 are met (for example, the internal components are within the field of view of the AR mobile device 12, within a threshold distance, and / or the cabinet 500 is selected by the user), and the cabinet 500 is determined to contain an obscured object / node (and as described above, for example, based on a known relationship between the cabinet and the internal components, and / Alternatively, by scanning a QR code, for example, the visual enhancement module 50 in Figure 1 can overlay the user interface 520 onto the user's view of the real world.

[0115] The user interface 520 includes a rendering 522 of the internal components. The rendering 522 may be an image of the actual components within the cabinet 500 in their actual arrangement relative to each other. In other embodiments, the rendering 522 includes a surreal or pseudo-realistic graphical representation of the components (e.g., a 3D model, a 2D model, etc.) individually or to show their relative arrangement within the cabinet 500. In any case, the rendering 522 may provide the user with an AR experience similar to the user's experience of viewing the actual internal components.

[0116] The user interface 520 may be directly superimposed on the cabinet 500 in the user's real-world view and appropriately scaled to align the drawing 522 over the actual positions of the hidden internal components. Alternatively, the user interface 520 may be offset from the cabinet 500 in the user's real-world view, possibly connected by lines or some other means.

[0117] The drawing 522 may or may not include node markers for individual internal components (e.g., similar to those in user interface 220 in Figure 4B) to the extent that those components have been previously mapped. Furthermore, user interface 520 may allow the user to select different node experiences by selecting such node markers, by selecting the drawing 522 itself, or by issuing an appropriate voice command. The drawing 522 or user interface 520 may also include a control (e.g., an "X" at one corner) that allows the user to choose to dismiss the drawing 522 from their view, or an appropriate voice command may be used.

[0118] Referring here to Figure 7A, the orientation indicator 550 includes a drawing of a specific object (in this case, a sensor / transmitter device) and an arrow indicating the general orientation of the object relative to the user's real-world view. The visual enhancement module 50 in Figure 1 may overlay the orientation indicator 550 on the user's real-world view in response to any of the conditions described above (relative to Figure 1), for example, with respect to avatars of remote and / or restricted-access objects. The drawing of the device in the exemplary orientation indicator 550 is a realistic model or image, but other formats are, of course, also possible (e.g., outlines, animations, videos, etc.).

[0119] If the user selects the direction indicator 550, the visual enhancement module 50 may overlay an "avatar" 560 onto the user's real-world view, as shown in Figure 7B. The avatar 560 shown in Figure 7B is identical to the rendering of the object in the direction indicator 550, although in other embodiments, the avatar 560 is more detailed and / or realistic than the rendering in the direction indicator 550 (if any). Similar to the rendering 522 in Figure 6, the avatar 560 may provide the user with an AR experience similar to viewing a real object (in this case, a sensor / transmitter device) up close. In fact, as described above, the virtual X-ray vision feature in Figure 6 could simply be a special case of the visual enhancement shown in Figure 7B, where the rendering 522 is a collection of avatars of various components enclosed within the cabinet 510.

[0120] When the user selects an avatar 560 or issues an appropriate voice command, the visual enhancement module 50 may overlay the user interface 570 near the avatar 560 in the user's real-world view. Alternatively, the user interface 570 may, without requiring any user selection of the avatar 560 or any voice command, In a fault, avatar 560 may be presented. The exemplary user interface 570 in Figure 7B provides process values ​​(here, sensor readings) and corresponds to a node experience that enables user control of the device (here, by initiating a calibration process). To enable this node experience, the experience provisioning unit 62 in Figure 1 may communicate with one or more applications 19 running on other servers 18, as described above. For example, the experience provisioning unit 62 may launch a process control application or interact with a running process control application using an API or the like.

[0121] Exemplary 3D mapping method Figure 8 illustrates an example method 600 for mapping a real-world process control environment (e.g., process control environment 100 in Figure 2) using a mobile device. Method 600 can be performed, for example, using the AR mobile device 12 in Figure 1 or another suitable mobile device. In some embodiments, a mobile device without AR capabilities may be used.

[0122] In block 602, the mobile device is registered at a reference or “landmark” location within the process control environment. Registration may include using the mobile device’s camera (e.g., camera 24 in Figure 1) to scan a visual representation of a code (e.g., a QR code or other image) located at the reference location. The code may be, for example, on a wall (e.g., a plate or poster).

[0123] In block 604, when the mobile device is moved through the process control environment by the user, at least one IMU of the mobile device (e.g., IMU 26 in Figure 1) is used to track the 3D position of the mobile device relative to a reference position and to track the orientation of the mobile device. Block 604 may continue indefinitely (e.g., simultaneously through blocks 606 to 610) until the user terminates the AR experience (e.g., by turning off the mobile device or closing an application running on the mobile device). Tracking in block 604 may also use the mobile device camera. For example, VIO technology may be used to track the position and orientation of the mobile device.

[0124] In block 606, user input is detected on the mobile device indicating that a new node should be added to the 3D map of the process control environment. User input may be, for example, a user voice command (e.g., "add node") or a user selection of a menu option (e.g., by focusing the mobile device camera on a specific location or by entering data on a keyboard or touchscreen).

[0125] In block 608, the 3D position of a real-world object (relative to a reference position) is determined or made to be determined based on at least the 3D position of the tracked mobile device and the orientation of the tracked mobile device. In one embodiment, block 608 includes capturing an image of a real-world object using the camera of the mobile device (for example, when user input is entered or immediately thereafter). The distance between the mobile device and the object can then be determined from the image by processing the camera image using at least depth sensing techniques. The 3D position can be determined or made to be determined based on the tracked 3D position, the orientation of the mobile device, and the determined distance.

[0126] In some embodiments, in block 608, the 3D position of the object is directly determined by a device, system, etc., performing method 600. In other embodiments, A device or system that implements method 600 may, for example, make a request via a web service interface (e.g., the web service interface 60 in Figure 1) to cause another device or system to determine the 3D position of an object.

[0127] In block 610, the node database or library can add a new node to the 3D map by storing at least the 3D position of an object in association with the new node. Block 610 may include, for example, using a web service interface to have a remote server (e.g., backend server 14 in Figure 1) store at least the 3D position of the new node in the node database (e.g., AR database 72 in Figure 1).

[0128] Method 600 may include one or more blocks not shown in Figure 8. For example, Method 600 may include a block in which another user input (e.g., a voice command, keyboard input, etc.) is received on a mobile device and the user input indicates an identifier (e.g., a name) and / or description of a new node. In such an embodiment, block 610 may further include storing the identifier and / or description of the new node in association with the new node in the node database.

[0129] As another example, method 600 may include additional blocks corresponding to subsequent actions of different mobile devices (or subsequent actions by the same mobile device performing blocks 602 to 610) navigating an area of ​​the process control environment after a new node has been added to the 3D map. In the first block, for example, a second mobile device (e.g., AR mobile device 12 or similar device) may be registered at a reference position. The 3D position (relative to the reference position) and orientation of the second mobile device may then be tracked using at least the IMU of the second mobile device (e.g., using VIO technology). Trigger conditions may then be detected by determining, for example, that an object is within the field of view of the camera of the second mobile device (e.g., based on the tracked orientation of the second mobile device) and / or that physical object is in proximity to the second mobile device (e.g., based on the tracked position of the second mobile device). In response to the detection of a trigger condition, digital information associated with a new node may be overlaid on or through the display of a second mobile device onto a view of the real world (e.g., node markers, or according to any node experience described above).

[0130] A method for providing an exemplary virtual X-ray vision Figure 9 shows an exemplary method 620 for providing virtual enhanced vision (e.g., virtual "X-ray vision") to a user of an AR mobile device in a real-world process control environment (e.g., process control environment 100 in Figure 2). Method 620 may be performed by one or more processors in a computing system or device, such as the backend server 14 in Figure 1, and / or by an AR mobile device, such as the AR mobile device 12 in Figure 1.

[0131] In block 622, it is determined that a specific node associated with the process control environment map ("first node") corresponds to a specific real-world object ("first object") currently within the field of view of the AR mobile device's camera (e.g., camera 24 in Figure 1). The first object may be, for example, any device illustrated or described in relation to Figure 2, or any other physical object that is at least partially opaque. In some embodiments, the first object is an enclosure, such as a cabinet, configured to contain a number of smaller devices or components.

[0132] In some embodiments, block 622 includes detecting the current orientation of the AR mobile device and determining, based on the current orientation of the AR mobile device and further based on the position of a first node indicated by the map, that a first object is currently within the camera's field of view. Block 622 may also include determining, based on the current position of the AR mobile device and the position of the first node, that the first object is within some threshold distance of the AR mobile device (e.g., 10 meters, 50 meters, etc.).

[0133] In block 624, a specific relationship is determined between a first node and one or more other nodes, which are also associated with the map. This relationship indicates (explicitly or implicitly) that one or more other real-world objects corresponding to one or more other nodes are at least partially obscured by the first object. For example, a node database (e.g., AR database 72 or a part thereof) may store data indicating that the first node corresponds to a cabinet or other enclosure containing one or more objects corresponding to other nodes (e.g., controllers, I / O cards, switches, etc.), and block 624 may include accessing the node database directly (e.g., in local memory) or via another computing system. In some embodiments, the first object is marked with a QR code or other visual code, and block 624 includes both scanning the code to determine the node identifier and using the node identifier to determine the relationship (e.g., by using the identifier as a key to the database).

[0134] In block 626, one or more digital models or digital images are retrieved from memory (e.g., local or remote persistent memory) to render an object in which the model(s) or image(s) are partially opaque by at least one object(s). For example, one or more surreal or pseudo-realistic 3D models, 2D models, animated models, digital images, or digital videos of objects(s) may be retrieved from memory.

[0135] In block 628, the display of the AR mobile device (e.g., display 22 in Figure 1) presents the retrieved model(s) or image(s) to the user while the first object is still within the camera's field of view. For example, the model(s) or image(s) may be presented to the user on the display or superimposed on a real-world view seen by the user through the display. Block 628 may include, for example, sending at least some of the display content and / or display commands to the AR mobile device (e.g., via the web service interface 60 in Figure 1).

[0136] Method 620 may include one or more blocks not shown in Figure 9. For example, Method 620 may include a block in which one or more node experiences are presented to the user about an obscure object (e.g., initially with a model(s) or image(s), or in response to further actions, such as the user selecting a model or image of a particular obscure object).

[0137] An exemplary method for visualizing restricted access objects Figure 10 illustrates an example method 640 that facilitates interaction between a user of an AR mobile device and a real-world object ("first object") that may be present in a restricted access area (e.g., a dangerous barricade) of a process control environment (e.g., process control environment 100 in Figure 2). Method 640 involves one or more processors of a computing system or device, such as the backend server 14 in Figure 1, and / or This can be performed by an AR mobile device such as the AR mobile device 12 shown in Figure 1.

[0138] In block 642, as the user moves through the process control environment, the display of the AR mobile device (e.g., display 22 in Figure 1) overlays digital information onto the portion of the environment within the field of view of the camera of the AR mobile device (e.g., camera 24 in Figure 1). The overlaid information is associated with nodes in the process control environment map (e.g., 3D map), along with nodes corresponding to other real-world objects in the environment. Block 642 may include, for example, sending at least some of the display content and / or display commands to the AR mobile device (e.g., via the web service interface 60 in Figure 1).

[0139] In block 644, while the first object is not in the camera's field of view, the AR mobile device's display indicates the direction to the first object. The indicator may include, for example, text, arrows, lines, and / or animations. Similar to block 642, block 644 may include, for example, transmitting at least some of the display content and / or transmitting display commands to the AR mobile device (e.g., via the web service interface 60 in Figure 1).

[0140] In block 646, after detecting user input (e.g., a voice command requesting the user interface of the first object) via an AR mobile device indicating the selection of a first object, the display superimposes a digital model or image of the first object (i.e., an "avatar" of the first object) and the user interface of the first object onto a portion of the process control environment that is currently in the field of view but does not contain the first object. The avatar may be, for example, a surreal or pseudo-realistic 3D model, 2D model, animated model, digital image, or digital video of the object. The user interface may include one or more virtual controls that enable user control of the first object (e.g., calibrating the first object, setting the operating parameters of the first object), and / or one or more displays that show information associated with a state (e.g., diagnostic state, maintenance state, operating state, etc.), settings of the first object, and / or outputs (e.g., sensor readings, etc.). In some embodiments, the user interface is not presented simultaneously with the avatar, but appears (along with the avatar) in response to a user selection of the avatar or a voice command.

[0141] In some embodiments, the directional indicator in block 644 occurs before detecting user input indicating the selection of a first object (i.e., the sequence shown in Figure 10). In some such embodiments, the directional indicator (e.g., an arrow pointing to an off-display area containing the first object) may be generated and displayed in response to detecting (1) that the first object is within a threshold distance of the AR mobile device, and (2) that the first object is not currently within the camera's field of view. Other suitable triggers or criteria for the directional indicator are also possible. For example, the indicator may be provided only to users having a specific user role associated with the first object.

[0142] In other embodiments, directional indicators arise after detecting user input indicating the selection of a first object. For example, the user may issue a voice command requesting the user interface to search for the first object (e.g., a user "search" for the first object), after which directional indicators (and possibly avatars) are superimposed on the user's real-world view.

[0143] Whether the digital model or image is displayed in the direction indicator and / or subsequent stages Regardless, the database can be retrieved using a web service interface (for example, using the web service interface 60 from the AR database 72 in Figure 1).

[0144] General Considerations When implemented in software, any of the applications and functions described herein may be stored as instructions in any tangible, non-transient, computer-readable memory, such as magnetic disks, laser disks, semiconductor memory devices, molecular memory storage devices, or other storage media, within the RAM or ROM of a computer or processor. While the exemplary systems disclosed herein are disclosed to include other components, software, and / or firmware running on the hardware, it should be noted that such systems are merely illustrative and not limiting. For example, any or all of these hardware, software, and firmware components may be embedded in hardware only, software only, or in any combination of hardware and software. Thus, although the exemplary systems described herein are described to be implemented in software running on the processor of one or more computer devices, those skilled in the art will readily understand that the examples provided are not the only ways of implementing such systems.

[0145] Therefore, although the present invention has been described in relation to specific examples, these examples are merely illustrative and not intended to limit the present invention, and it will be apparent to those skilled in the art that modifications, additions, or deletions may be made to the disclosed embodiments without departing from the spirit and scope of the invention.

Claims

1. A method for mapping a real-world process control environment using a mobile device, wherein the method is Registering the mobile device at a reference position within the process control environment, Track (i) the position of the mobile device relative to the reference position, and (ii) the orientation of the mobile device, using at least the inertial measurement unit of the mobile device. The mobile device detects user input indicating that a new node should be added to the three-dimensional (3D) map of the process control environment, The distance between the mobile device and the real-world object is determined by processing images of the real-world object using at least depth sensing technology, At a minimum, the 3D position of the real-world object relative to the reference position is determined or the 3D position of the real-world object is determined based on (i) the tracked position of the mobile device, (ii) the orientation of the tracked mobile device, and (iii) the distance between the mobile device and the real-world object. At least the 3D position of the real-world object is associated with the new node and stored in the node database, thereby adding the new node to the 3D map of the process control environment, A method that includes this.

2. Registering the mobile device at the aforementioned reference position This includes scanning a visual representation of a code using the camera of the mobile device, wherein the visual representation is located at the reference position. The method according to claim 1.

3. Using the camera of the aforementioned mobile device, a visual representation of the code can be scanned. This includes scanning a QR code located at the reference position using the camera of the mobile device. The method according to claim 2.

4. Tracking the position and orientation of the mobile device This includes tracking the position and orientation of the mobile device using the inertial measurement unit and the camera of the mobile device. The method according to claim 1.

5. Tracking the location of the mobile device This includes tracking the 3D position of the aforementioned mobile device. The method according to claim 1.

6. Detecting user input indicating that a new node should be added to the 3D map, The method according to claim 1, comprising detecting voice commands, keyboard input, or touchscreen input.

7. Determining the 3D position of the real-world object, or causing the 3D position of the real-world object to be determined, This includes capturing the image of the real-world object using the camera of the mobile device, The method according to claim 1.

8. The mobile device further includes receiving user input indicating an identifier and / or description of the new node, Adding the new node to the 3D map of the process control environment in the node database further includes storing the identifier and / or description of the new node in association with the new node. The method according to claim 1.

9. The method according to claim 1, wherein the mobile device is an augmented reality (AR) mobile device that overlays digital information onto a view of the real world presented through the display of the AR mobile device.

10. A system for mapping a process control environment in the real world, wherein the system Server and A mobile device equipped with an inertial measurement unit, wherein the mobile device is Registered at a reference position within the aforementioned process control environment, Using at least the inertial measuring unit, (i) the position of the mobile device relative to the reference position, and (ii) the orientation of the mobile device, A mobile device is configured to detect user input indicating that a new node should be added to the three-dimensional (3D) map of the process control environment, Either the mobile device or the server determines the distance between the mobile device and the real-world object by processing an image of the real-world object using at least depth sensing technology. (i) the mobile device or the server that is tracked The system is configured to determine the 3D position of the real-world object relative to the reference position based on the position of the device, (ii) the orientation of the tracked mobile device, and (iii) the distance between the mobile device and the real-world object. The server is configured to add a new node to the 3D map of the process control environment, which stores the 3D position of the real-world object associated with the new node in a node database. system.

11. The aforementioned mobile device further includes a camera, The mobile device is configured to register at the reference position by scanning a visual representation of a code using at least the camera, and the visual representation is located at the reference position. The system according to claim 10.

12. The aforementioned mobile device includes at least, It is configured to detect user input indicating that a new node should be added to the 3D map by detecting voice commands, keyboard input, or touchscreen input. The system according to claim 10.

13. The aforementioned mobile device further includes a camera, The mobile device is configured to capture the image of the real-world object using the camera. The system according to claim 10.

14. The system according to claim 10, wherein the mobile device is an augmented reality (AR) mobile device further comprising a display that overlays digital information onto a view of the real world.

15. Tracking the location of the mobile device This includes tracking the 3D position of the aforementioned mobile device. The system according to claim 10.

16. By being executed by one or more processors, the one or more processors Registering a mobile device at a reference location within the process control environment, Track (i) the position of the mobile device relative to the reference position, and (ii) the orientation of the mobile device, using at least the inertial measurement unit of the mobile device. The mobile device detects user input indicating that a new node should be added to the three-dimensional (3D) map of the process control environment, The distance between the mobile device and the real-world object is determined by processing images of the real-world object using at least depth sensing technology, At a minimum, the 3D position of the real-world object relative to the reference position is determined or the 3D position of the real-world object is determined based on (i) the tracked position of the mobile device, (ii) the orientation of the tracked mobile device, and (iii) the distance between the mobile device and the real-world object. By associating and storing at least the 3D position of the real-world object with the new node, the node database can store the 3D map of the process control environment. Adding the aforementioned new node to the program, One or more non-temporary computer-readable media on which instructions for executing a command are stored.

17. Registering the mobile device at the aforementioned reference position This includes scanning a visual representation of a code using the camera of the mobile device, wherein the visual representation is located at the reference position. One or more non-temporary computer-readable media according to claim 16.

18. Using the camera of the aforementioned mobile device, a visual representation of the code can be scanned. This includes scanning a QR code located at the reference position using the camera of the mobile device. One or more non-temporary computer-readable media according to claim 16.

19. Tracking the position and orientation of the mobile device This includes tracking the position and orientation of the mobile device using the inertial measurement unit and the camera of the mobile device. One or more non-temporary computer-readable media according to claim 16.

20. Determining the 3D position of the real-world object, or causing the 3D position of the real-world object to be determined, This includes capturing an image of the real-world object using the camera of the mobile device, One or more non-temporary computer-readable media according to claim 16.

21. The mobile device further includes receiving user input indicating an identifier and / or description of the new node, Adding the new node to the 3D map of the process control environment in the node database further includes storing the identifier and / or description of the new node in association with the new node. One or more non-temporary computer-readable media according to claim 16.

22. The mobile device is an augmented reality (AR) mobile device that overlays digital information onto a view of the real world presented through the display of the AR mobile device, one or more non-temporary computer-readable media according to claim 16.

23. Tracking the location of the mobile device This includes tracking the 3D position of the aforementioned mobile device. One or more non-temporary computer-readable media according to claim 16.

Images