Augmented reality-based automation systems

Abstract

An automation system for controlling an automated production process of an industrial plant is provided. The system includes an augmented reality system including augmented reality glasses and executable program logic coupled to the augmented reality glasses. The executable program logic is configured to receive a current spatial position of a user wearing the augmented reality glasses from a positioning system, determine coordinates of a bounded space proximate to the user wearing the AR glasses, display one or more virtual objects to the wearer of the AR glasses via the AR glasses if and only if the coordinates of the virtual objects are within the bounded space, and continuously update the coordinates of the three-dimensional bounded space and the virtual objects displayed therein according to the current spatial position of the user wearing the AR glasses.

Description

[Technical field]

[0001] Certain embodiments of the present invention relate to the field of automation systems, more particularly to augmented reality based automation systems applied in industrial plants. [Background technology]

[0002] In industrial plants, input materials are processed to produce one or more products, and therefore the characteristics of the produced products depend on the production parameters and the input materials.

[0003] Within the process industry, and particularly in industrial plants such as chemical or biological production plants, one or more input materials are processed using a production process to produce one or more chemical or biological products. The production environment in the process industry can be complex, and thus the properties of the product may vary according to the variations in the production parameters that affect said properties. Typically, the dependence of the properties on the production parameters is complex and may be intertwined with further dependence on one or more combinations of specific parameters. Thus, it may be difficult to produce chemical or biological products with consistent and / or predictable quality.

[0004] In particular, product quality and / or production stability depend on operators having quick, easily accessible access to the operating characteristics of the plant equipment and supporting infrastructure, as well as up-to-date information regarding the state of the plant, including values ​​of current process parameters corresponding to each piece of equipment, values ​​of ambient environmental plant parameters, and / or values ​​of parameters associated with raw materials and intermediate products produced during processing of materials. Information regarding the state of the plant may also include the maintenance history of the equipment and information regarding how each piece of equipment is operating, including ranges of operating parameters for safely operating the equipment.

[0005] The operation and maintenance of production equipment typically requires specialized knowledge of machines and manufacturing processes. However, employees with the appropriate knowledge are often in short supply. At the very least, when a machine has a technical problem, it can take time to get properly trained maintenance personnel on-site.

[0006] Thus, there is a need for techniques that can improve the process of controlling and / or maintaining automated plant manufacturing processes to provide products of consistent and / or predictable quality.

[0007] Summary of the Invention At least some of the problems inherent in the prior art are shown to be solved by the subject matter of the attached independent claims. At least some of the further advantageous alternatives are outlined in the dependent claims.

[0008] Various embodiments provide a system for controlling an automated production process in an industrial plant as described in the subject matter of the independent claims. Advantageous embodiments are set out in the dependent claims. The embodiments of the invention can be freely combined with one another if they are not mutually exclusive.

[0009] In one aspect, the invention relates to an automation system for controlling an automated production process of an industrial plant, the automation system including an augmented reality system, the augmented reality system including augmented reality glasses and executable program logic coupled to the augmented reality glasses, the executable program logic configured to receive a current spatial position of a user wearing the augmented reality glasses from a positioning system, dynamically determine coordinates of a boundary space proximate to the user wearing the AR glasses, display one or more virtual objects to the wearer of the AR glasses via the AR glasses if and only if the coordinates of the virtual objects are within the boundary space, the displayed virtual objects comprising data associated with monitoring and / or control of the automated production process, and continuously update the boundary space and the coordinates of the virtual objects displayed in the boundary space according to the current spatial position of the user wearing the AR glasses.

[0010] According to an embodiment, the executable program logic is configured to receive data indicating an orientation of the AR glasses and is configured to dynamically determine and continuously update coordinates of the boundary space in response to the data indicating the orientation of the AR glasses.

[0011] According to an embodiment, the executable program logic is configured to receive data indicative of an eye orientation of a user wearing the AR glasses, and configured to dynamically determine and continuously update coordinates of the boundary space in response to the eye orientation of the user wearing the AR glasses.

[0012] These features have the advantage of allowing a user (i.e., wearer) of the AR glasses to quickly and efficiently obtain the latest virtual objects contained within the bounded space that changes dynamically according to the user's variable spatial position and / or the variable orientation of the AR glasses, as well as the advantage of defining (i.e., "filtering") those objects for display by using the bounded space defined via the determined coordinates. Furthermore, the virtual objects represented by data such as text, image, video, alarm, and / or audio data overlay (i.e., augment) the view of the plant through the AR glasses, thereby enabling the user to efficiently control and / or maintain the automated process in a hands-free manner (i.e., the user does not have to carry instruction manuals and maintenance records when inspecting and managing the production line and production process). Furthermore, according to an embodiment, the executable program logic is configured to limit the data (i.e., virtual objects) displayed to the user through the augmented reality glasses (AR glasses) to only those virtual objects contained within the bounded space, so that the amount of data does not overwhelm the user.

[0013] In an embodiment, the executable program logic is configured to dynamically determine coordinates of the boundary space depending on a number of equipment and / or a corresponding number of virtual objects comprising data of said equipment in spatial proximity to the user wearing the AR glasses and / or depending on dimensions of a plant infrastructure in proximity to the user, such as a height of a room and / or compartment in which a latest spatial position of the user wearing the AR glasses is located and / or depending on one or more user roles assigned to the user wearing the AR glasses and / or depending on whether the boundary space is defined to be a data type specific boundary space, wherein the boundary space is defined to include only virtual objects comprising objects of one or more data types selected from the group consisting of video, image, text, audio file, and alarm message.

[0014] According to some embodiments, the size and / or shape of the bounding space is determined depending on the data type of the virtual object to be displayed. For each data type, a respective data type specific bounding space may be generated and used to determine whether or not a particular virtual object should be output via the AR glasses.

[0015] A compartment can be, for example, a cavity associated with an equipment, such as a tank, or a cavity of a machine. A user may need to enter the compartment for maintenance purposes. In this case, it may be useful for the bounding space to reflect the interior dimensions of the compartment to avoid overloading the user with virtual objects containing information of nearby machines or machine parts that the user cannot operate in any case in his current position.

[0016] For example, a compartment may be labeled with a QR code or an RFID tag or similar tag that includes a room ID assigned to this compartment. The AR glasses or another electronic device of the user, for example the user's smartphone, may be equipped with a reading device (camera, RFID reader, etc.) configured to read the room ID from the tag or label when the user wearing the AR glasses enters the compartment. The room ID is transferred to the executable program logic, enabling the program logic to determine the dimensions of the compartment as entered by the user and to create and use a bounded space, whose coordinates are determined according to the compartment's dimensions or ID.

[0017] According to an embodiment, the executable program logic is configured to dynamically determine the coordinates of the boundary space depending on the number of devices and / or depending on the corresponding number of virtual objects comprising data of said devices in spatial proximity to the user wearing the AR glasses. The dynamic determination of the coordinates of the boundary space includes in particular determining the size and / or shape depending on the number of devices or on the number of corresponding virtual objects.

[0018] These features, alone or in different combinations with one another, have the advantage of providing a boundary space that is specifically tailored to provide a refined presentation of virtual objects to the user, such as, for example, providing a boundary space that includes only the most urgent virtual objects to display to the user given the user's current current spatial location and / or orientation of the AR glasses, or, for example, providing a boundary space that includes only the virtual objects most relevant to one or more roles assigned to a particular user, including optionally displaying relevant virtual objects that are not necessarily in close proximity to the user, such as virtual objects corresponding to equipment located on different floors and / or rooms or areas of the plant that cannot be displayed to the user without adjusting the boundary space. Furthermore, providing a boundary space whose size and shape are adjustable results in the user not being overwhelmed with a very large number of virtual objects, particularly in connection with providing a boundary space to a user of AR glasses whose spatial location is within a high equipment density area of ​​the plant.

[0019] In another aspect, the present invention relates to a method for controlling an automation system for controlling an automated production process of an industrial plant, the automation system comprising an augmented reality system including augmented reality glasses and executable program logic coupled to the augmented reality glasses, the method comprising: receiving, by the executable program logic, a current spatial position of a user wearing the augmented reality glasses from a positioning system; dynamically determining, by the executable program logic, coordinates of a boundary space proximate to the user wearing the AR glasses; displaying, by the executable program logic, one or more virtual objects to the wearer of the AR glasses via the AR glasses if and only if the coordinates of the virtual objects are within the boundary space, the displayed virtual objects comprising data associated with monitoring and / or controlling the automated production process; and continuously updating, by the executable program logic, the coordinates of the three-dimensional boundary space and the coordinates of the virtual objects displayed in the three-dimensional boundary space as a function of the current spatial position of the user wearing the AR glasses and / or as a function of the orientation of the AR glasses and / or as a function of the user's eye orientation ("gaze").

[0020] In one aspect, the present invention relates to an automation system for controlling an automated production process of an industrial plant, the industrial plant including a plurality of pieces of equipment for performing the production process, the plurality of pieces of equipment being spatially distributed within the industrial plant. The automation system includes an augmented reality system including a database system configured to store data and spatial coordinates associated with each piece of equipment of the plurality of pieces of equipment, augmented reality glasses configured to display the data, and executable program logic coupled to the augmented reality glasses. The executable program logic is configured to receive from a positioning system a current spatial location of a user wearing the augmented reality glasses and data of the equipment located proximate to the user's current spatial location from the database system. The executable program logic is further configured to control the augmented reality glasses to display at least a portion of the received data.

[0021] These features have the advantage of quickly, efficiently, and securely receiving data by a user of the AR glasses that corresponds to equipment in the user's vicinity, and the data overlays (i.e., augments) a view of the plant through the AR glasses, thereby allowing the user to efficiently control the automated process in a hands-free manner (i.e., the user does not need to carry instruction manuals or maintenance records when inspecting and controlling the production line and production process). Furthermore, according to an embodiment, the executable program logic is configured to limit the data displayed to the user via the augmented reality glasses (AR glasses) to data assigned to equipment in the vicinity of the user's location, so that the amount of data does not overwhelm the user. The executable program logic is further configured to control the augmented reality glasses to display at least a portion, if not all, of the received data.

[0022] In one embodiment, the augmented reality system (AR system) further includes a positioning system. The positioning system is configured to detect the spatial location of the user within the plant. These features have the advantage of tailoring the positioning system to the requirements of an industrial plant. For example, if the plant is compartmentalized rather than an open floor plan, commercially available GPS systems may not be effective in such a closed indoor environment. The positioning system as a component of the augmented reality system may alleviate such issues.

[0023] For example, according to further embodiments, the positioning system may include a spatial mapping mesh covering at least a portion of the plant, an indoor positioning system, a cellular positioning system, or even a combination of one of the preceding systems with a GPS system.

[0024] According to an embodiment, the spatial mapping mesh comprises spatial anchors, the spatial anchors being interconnected, each spatial anchor relating a coordinate system associated with the augmented reality glasses to a spatial coordinate system associated with the spatial coordinates of one or more of the devices stored in the database system for displaying data, the program logic being configured to display data of a device located in proximity to the user's current spatial position via the augmented reality glasses at a predetermined relative position to the spatial anchor.

[0025] According to an embodiment, the augmented reality system is configured to create an avatar of at least one remote user. The remote user is a user remote from the industrial plant. The program logic is configured to display the avatar via the augmented reality glasses in proximity to the most recent spatial location of the user wearing the augmented reality glasses. According to a preferred embodiment, the augmented reality system is configured to provide a two-way visual and / or acoustic communication channel between the user wearing the augmented reality glasses and the remote user. The AR glasses thereby give the wearer the impression that the wearer is speaking directly to the avatar and not to the remote user. For example, the program logic displays the avatar via the AR glasses and / or controls the output of the remote user's voice via the speakers of the AR glasses such that the position of the avatar is perceived as the source of the remote user's voice.

[0026] According to some embodiments, the automation system comprises a robot having a camera and / or microphone. The robot is located in the industrial plant and is therefore "local" to a user wearing the AR glasses and working in the industrial plant. The augmented reality system comprises a robot control module configured to control the movement and / or orientation of the robot such that the position and / or orientation of the robot reflects the position and / or orientation of an avatar in the augmented reality displayed through the AR glasses. The robot control module is configured to automatically update the position and / or orientation of the avatar and the robot according to changes in the position and / or orientation of the remote user or in response to control commands sent by the remote user.

[0027] This allows the "reality" of remote users, local users, and various physical objects (raw materials, equipment) to be seamlessly integrated and merged into a single augmented reality displayed through the AR glasses.

[0028] According to an embodiment, the augmented reality glasses include an audio output interface, and the augmented reality system is configured to provide a bidirectional audio communication channel between a user wearing the augmented reality glasses and an avatar, and the program logic is configured to control the audio output interface such that a volume of the avatar's voice output to the user wearing the augmented reality glasses via the audio output interface is negatively correlated with a distance between the user wearing the glasses and the avatar in a coordinate system associated with the augmented reality glasses displaying the data.

[0029] The audio output interface may be, for example, a speaker in the environment of the user wearing the AR glasses, a headphone that may be integrated into the AR glasses, or a built-in speaker.

[0030] In one embodiment, the database system is a graph database system, and / or a spatial database system, and / or a streaming database system.

[0031] In another embodiment, the automation system further includes a control system. The control system can be a distributed control system. The control system includes a memory storing one or more process parameters and a control software. At least some of the parameters are stored in association with one or more of the equipment, for example by storing the parameters in association with an equipment ID. The control system comprises a processor configured to execute a process control software for automatically controlling a production process. For example, the control software can control the use of the equipment such that the production process is performed according to one or more parameters assigned to said equipment. For example, this can mean that one or more of the parameters represent a desired machine state and / or product characteristic, and the control is performed such that the machine operates in the desired state and / or the product has the desired characteristic. For example, if the product has a desired shape, size and / or temperature, the production process can be controlled such that the difference between the measured and observed product characteristic and the desired product characteristic is minimized.

[0032] According to an embodiment, the AR system is configured to generate and display a GUI via the AR glasses that enables a user wearing the AR glasses to set one or more of the parameters, for example via gesture detection and gesture interpretation.

[0033] For example, the AR glasses may be equipped with a gyroscope configured to detect the orientation of the AR glasses and of the user carrying the AR glasses. Additionally, the AR glasses may be equipped with one or more cameras configured to capture one or more images in front of and near the user. The executable program logic may be configured to analyze the digital images captured by the cameras of the AR glasses to identify the user's gestures. The program logic may be configured to identify the user's actions and input, modification, or deletion of parameters via the GUI depending on the type of gesture, the location and / or the direction of the gesture.

[0034] These features have the advantage that, based on the data communicated to the user via the AR glasses, the user of the AR glasses may modify the control software of the distributed control system and change the operation of the equipment to provide a consistent and predictable product. For example, the user may determine that a particular sensor of the equipment or equipment part is too sensitive and may falsely detect production errors (false positive problem). In this case, the user may set the sensitivity control parameter of this sensor to a low value, thereby reducing the number of false positive alarms that may slow down the production process. According to another example, the user may determine that an actuator of the equipment is too weak, thereby causing an unacceptably large number of defective products. In this case, the user may use the GUI presented via the AR glasses to increase a parameter that controls the power / operation amplitude of the actuator of the equipment.

[0035] According to an embodiment, the AR system is configured to generate a GUI in which the user may selectively set, modify or delete parameters assigned to equipment that are within a predefined maximum distance from the user. For example, the maximum distance may be 5m, or 2m, or 1m. This may have the advantage that only parameters that are relevant when dealing with a particular equipment or equipment part are presented to the user.

[0036] According to an embodiment, the control system is a distributed control system comprising multiple control software programs or modules interconnected with each other via a network.

[0037] According to a further embodiment, the at least one piece of equipment comprises one or more sensors coupled to the at least one piece of equipment and one or more actuators coupled to the at least one piece of equipment. The one or more sensors are configured to measure one or more process parameters of the at least one piece of equipment, the one or more process parameters being indicative of a current state of the production process of the plant and / or indicative of a current state or operation mode of the equipment. The one or more actuators are configured to control the one or more process parameters. Further, a distributed control system is coupled to the one or more sensors and the one or more actuators, and a processor is configured to execute process control software for automatically controlling the production process based on measurement signals received from the one or more sensors and control signals sent to the one or more actuators. Additionally or alternatively, the one or more actuators are configured to operate according to one or more control parameters (e.g., pressure, temperature light intensity, concentration of one or more chemicals, machine operation mode, etc.).

[0038] According to further embodiments, the claimed automation system includes a user database coupled to the distributed control system. The user database is configured to store user information associated with the production process. In some embodiments, the user information includes a user ID, a user role, and / or a user privilege.

[0039] In yet another embodiment, the executable program logic is configured to receive user credentials from a user and authenticate the user. For example, the authentication can be implemented as a function of matching the user credentials with one of a plurality of user identities. The user identity of the user wearing the augmented reality glasses is associated with one or more user roles, each of the one or more user roles including one or more user privileges. The executable program logic is further configured to select and display content of at least some of the received data for display based on the one or more user roles and / or the one or more user privileges associated with the user identity.

[0040] These features have the advantage of only providing data to authorized users (i.e., providing data via a secure system), and if the user is authorized, the selection of data viewed by the user, which may include the type of data and / or the content of the data, is further based on one or more user roles and / or one or more user privileges associated with the user's user ID.

[0041] According to an embodiment, the executable program logic is configured to generate a GUI that is displayed to the user via the AR glasses, whereby the content of the displayed GUI is dynamically determined depending on the role assigned to the user wearing the AR glasses. The executable program logic is configured to generate a first type of GUI for the user if the user is within a predetermined maximum distance from a particular equipment and has a first role, and to generate a second type of GUI for the user if the user is within a predetermined maximum distance from said particular equipment and has a second role. The first and second GUIs are different, e.g., comprise different GUI elements, display different information, and enable different types of actions for maintaining and / or controlling said equipment. For example, the first user role can be an "operator", i.e. a user who is obliged to control and / or monitor one or more steps of a production workflow. The second user role can be a "maintenance worker", i.e. a user who is obliged to perform maintenance actions to ensure that at least one equipment is operating properly. For example, the maintenance worker may need to repair a broken machine, replace a deteriorated part of the equipment, perform some quality test of the equipment, etc. Typically, a maintenance worker is responsible for technically complex tasks that may require detailed technical knowledge of the equipment and are not typically available to a user with the role of "operator." Thus, when an "operator" user approaches a particular equipment / machine, they will see a different GUI than the "maintenance worker" sees through the AR glasses when approaching that equipment / machine.

[0042] This allows various tasks to be performed efficiently and error-free with the aid of or on the equipment, since each user sees, via the AR glasses, only the information actually required to perform the actual task. Since the "operator" user does not "see" the GUI that may change the machine parameters, the risk of an error scenario in which a "normal operator" causes an error in the production process by mistakenly changing a machine parameter that should only be changed by a skilled maintenance worker, can be significantly reduced.

[0043] According to an embodiment, the executable program logic is configured to perform a database query based on the most recent spatial location to retrieve from the database data on devices proximate to the user's most recent spatial location.

[0044] According to an embodiment, the database system is a spatial database system, the spatial database system including a spatial database, and the executable program logic is configured to perform a query of the spatial database based on the most recent spatial location to retrieve from the spatial database data for devices proximate to the most recent spatial location of the user.

[0045] These features have the advantage of storing, searching and retrieving data from the database system based on a query of the spatial database, including the most recent spatial location of the user of the AR glasses. That is, data associated with plant objects in close proximity to the spatial location of the user is linked to the spatial location of the plant objects that are also stored in the database system, thereby enabling a quick and efficient way of accessing the respective data when the database is queried. These features also have the advantage of retrieving data only when a query is required, thereby saving network bandwidth.

[0046] In another embodiment, the database system is accessible via a subscription-based streaming service. This database system may be referred to as a streaming database system. It may be implemented as a spatial and / or graph-based database system.

[0047] The executable program logic is configured to receive data of devices proximate to the user's most recent spatial location from the database via the streaming service. According to an embodiment, the executable program logic is configured to selectively receive data of devices proximate to the user's most recent spatial location from the spatial database via the streaming service (excluding data of devices not proximate to the user). For example, a predefined or user-defined maximum distance threshold may be used by the database system to determine whether a device's location is proximate to the user. These features have the advantage of quickly accessing data continuously via the streaming service, thereby eliminating the sending and / or receiving of queries.

[0048] For example, a user can subscribe to a streaming service that notifies the user when at least one device approaches the user and is then located within a predetermined maximum distance from the user. When the user subscribes to this service, the user's current location is automatically determined and provided as one parameter of the subscription request. The database system is configured to automatically and repeatedly check whether the distance between the user subscribing to the notification service and the at least one device is less than a maximum distance threshold. In response to this determination, the GUI, one or more GUI elements, and / or parameters assigned to the at least one device are displayed to the user via the AR glasses. The user's subscription to the service is automatically implemented by program logic, for example, when the user is successfully authenticated in the AR system and / or when the user approaches a production system closer than a user-defined or predetermined maximum distance, for example, less than 10 m, or less than 5 m, or less than 2 m, or less than 1 m.

[0049] According to an embodiment, the database system is configured to selectively provide data of devices proximate to a current spatial location of a user wearing the augmented reality glasses in response to a query (e.g., a conventional SQL query) of the executable program logic. Movement of the user relative to the device triggers the executable program logic to send a new query to the database system including the user's new current spatial location.

[0050] According to another embodiment, the database system is configured to selectively provide data of a device located proximate to a current spatial location of a user wearing the augmented reality glasses via a subscription-based streaming service, where movement of the user relative to the device triggers executable program logic to update the subscription with the user's new current spatial location.

[0051] According to an embodiment, the program logic is configured to automatically update the user's subscription to the streaming service in response to a change in the user's location. For example, the AR glasses may be equipped with a GPS sensor or another means for determining the user's current location. In response to determining that the user has changed his / her location, the user's subscription to the streaming service of the database system is automatically updated, thereby ensuring that the user can always view data and GUIs related to one or more devices next to the user. This may have the advantage that the AR system and its components do not need to repeatedly send conventional SQL queries with the user's current location information. In order to ensure that the data displayed via the AR glasses takes into account the user's current location, it is sufficient to update the user's subscription to one or more streaming services associated with the database system, thereby significantly reducing network traffic and the overall computational load of the AR system, in particular the computational load of the database system.

[0052] According to yet another embodiment, the spatial database system is a graph database system including a graph database having a plurality of interconnected nodes. Each node represents one or more objects of an industrial plant and includes spatial information associated with the one or more objects. For example, when a node represents multiple objects, these objects are usually in close proximity to each other during the manufacturing process. Depending on the embodiment, the spatial information associated with the one or more objects may be the location information of a single object of the multiple objects, or may be the location information of each of the multiple physical objects. The object is a sensor, an actuator, a raw material, a piece of equipment of the multiple equipment, e.g., a tool, or a machine or robot. The architecture of the graph database allows for fast search and access of data associated with a given object in the plant using only the spatial coordinates of a user of the AR glasses in the plant. Furthermore, by representing objects of the industrial plant that are in close proximity to each other during the manufacturing process by a single node in the graph, it is possible to monitor the location of at least approximately all objects of the node such that location information of at least one of the objects represented by the node is available.

[0053] According to some embodiments, the executable program logic is configured to perform the following: receiving coordinates of a visual object that graphically represents data of the device to be displayed, for example the program logic may receive coordinates, for example the 3D coordinates and size of a visual representation of a handbook or instruction manual for the device, or the coordinates of a virtual "TV screen" on which a tutorial for operating the device is displayed via the AR glasses; determining a line of sight of the user in a real-world coordinate system, for example the executable program logic may receive from a positioning system an indication of a current location and orientation of a user wearing the AR glasses; receiving from the database system coordinates of a real-world object in proximity to the most recent spatial position of the user wearing the augmented reality glasses, the real-world object being a portable and / or mobile device, a raw material, or a product; the real-world object may be an object whose current location is constantly monitored and tracked by the augmented reality system, for example by a GPS sensor, an indoor positioning system, one or more capacitive or inductive proximity sensors, etc., such that the coordinates of the real-world object indicate the actual position, orientation and size of the real-world object in the real world; for example, the program logic may receive coordinates of a virtual mesh generated as an overlay of equipment of a real-world production line, the received coordinates may also be coordinates dynamically determined by a positioning sensor for the equipment, or for another type of real-world object; determining whether one or more of the virtual objects representing data of the device are positioned along a line of sight of the user and are farther from the user than at least one of the real-world objects positioned along the line of sight of the user; and Selectively not displaying one or more virtual objects that are determined to be in the user's line of sight and located farther away than at least one of the real-world objects.

[0054] This can be advantageous if a virtual object, e.g. a handbook, is displayed, but is obscured by a "real world object" and it is decided not to display this virtual object.

[0055] According to an embodiment, the executable program logic comprises: receiving data indicative of a location of one or more physical objects, each physical object being, in particular, equipment used in an automated production process or material used, modified, or produced by the production process; Determining a line of sight between a user wearing the glasses and one of the virtual objects; Displaying one virtual object through the AR glasses if and only if the virtual object is within a bounded space and no physical objects are located within the line of sight between the user wearing the glasses and the virtual object; It is configured as follows.

[0056] This feature may prevent the user from being distracted, since it is assumed that real-world objects closer to the user on the user's line of sight (axis of gaze) than a hologram with, for example, an educational film, require greater attention. For example, the user may carry out repair work on a conveyor belt while the conveyor belt continues to operate. As soon as a workpiece transported on the belt enters the user's view direction and moves in front of the hologram, the hologram is completely or partially hidden (as the AR system stops displaying the hologram, or at least the part of the hologram hidden by the real-world object). This has the advantage that the user is always aware of the moving object, thus avoiding accidents. In addition, reading errors are also avoided, since when the hologram is overlaid on a dynamically changing real-world scene using AR glasses, the brightness and contrast of the background change, which may cause the user to make mistakes when reading the data of the virtual object. Additionally, real physical objects, which may be static or moving, can overlay and obscure virtual objects, making the visual impression more "immersive" for the user, as the dynamically "hidden" virtual objects appear realistically and blend seamlessly with the physical world objects.

[0057] According to an embodiment, one or more of the nodes comprise or are stored in association with a respective container.

[0058] A "robot" as used herein is a machine that can move and perform a specific task automatically. According to some embodiments, the robot comprises a robot control module configured to control the movement, position, orientation, and / or tasks performed by the robot. Additionally or alternatively, the robot can be operatively coupled to a robot control module external to the robot. For example, the control system described herein for embodiments of the invention can comprise a robot control module or a portion thereof. The robot control module is configured to receive state information from one or more sensors of the robot and transmit control commands to the robot or to one or more parts of the robot, whereby the control commands determine the movement, position, orientation, and / or tasks performed by the robot. The one or more sensors of the robot can comprise positioning sensors, such as GPS sensors, gyroscopes, and accelerometers, for determining the position and orientation of the robot, and cameras and microphones for capturing images or videos, and / or acoustic signals.

[0059] The coordinate system "associated with the AR glasses" is an augmented reality coordinate system that comprises real world objects (such as machines and raw materials) and one or more virtual objects. In the augmented reality coordinate system, the coordinates of the real world objects and the virtual objects are mapped to this augmented reality coordinate system and displayed to the user wearing the AR glasses via said glasses, such that the virtual objects and the real world objects form part of a shared space (called augmented reality). The coordinate system "associated with the AR glasses" can therefore be considered as an augmented reality coordinate system.

[0060] A "container" as used herein is an isolated runtime environment, e.g., an isolated user space instance. For example, a container may be an operation system level virtualization unit, where the kernel of the operation system enables and supports the existence of multiple isolated virtual runtime environments in parallel. For example, one of the currently available container platforms, such as LXC, Solaris containers, Docker containers, virtual private servers (OpenVZ), virtual environments (VEs), virtual kernels (DragonFly BSD), or jails (FreeBSD jails or chroot jails), can be used to implement a container. Each container may generate a runtime environment that makes the software programs running in them look like a real computer.

[0061] According to some embodiments, each container is stored within or assigned to a respective node of the graph of the graph database of the database system described herein. Each container comprises software programs and / or functions configured to monitor and / or control the position, orientation and / or motion of one or more physical objects (e.g., raw materials, equipment, etc.) represented by said node, rather than the physical objects represented by other nodes of the graph. Thus, each container may encapsulate the functionality used to monitor and / or control one or more objects. This may prevent unforeseen errors and conflicts that may be caused by unknown software interdependencies. For example, software problems may occur when a device driver for a machine is replaced with a new version, while there is another machine whose control relies on the old version of the driver.

[0062] Depending on the embodiment, the degree of isolation of containers from one another may vary to enable at least limited, well-defined, and therefore highly secure data exchange between the containers and the software contained in them.

[0063] According to some embodiments, each container implements an application programming interface (API) that allows the container to exchange data with other containers exclusively through this API. This can prevent situations where software dependencies between multiple devices are monitored, and can allow container-specific data exchange and control protocols that define which other containers can exchange what kind of data with this container through the API.

[0064] According to another embodiment, there may be a central data exchange interface provided at the application level by the container management software, which allows data exchange between containers in a centralized and informal manner, and may nevertheless prevent errors caused by hidden software interdependencies.

[0065] A computer program running on a typical operating system can "see" (detect the presence and / or access to) all of the resources of that computer (connected devices, files and folders, network shares, CPU power, quantifiable hardware capabilities). However, according to some embodiments of the invention, a program running inside a container can only "see" the contents of the container, and the devices assigned to the container. In this case, software within the container can only see objects represented by graph nodes assigned to the container. In some embodiments, containers are programmed and / or configured such that they can only exchange data with other containers (thus on the application layer) via one or more APIs provided by the respective container and / or central container management software.

[0066] For example, the one or more objects monitored and / or controlled by the software of a container may be or may comprise one or more sensors and / or actuators of equipment (e.g., tools, machines, robots), of equipment or raw materials. Thus, the software in the container can only "see" and optionally modify parameters of the equipment represented by the respective node of the container. Objects represented by other containers assigned to other nodes cannot be seen or manipulated unless there is a defined API for data exchange with this other container.

[0067] According to some embodiments, each of the containers comprises software for monitoring and / or operating one or more objects represented by said nodes, and further comprises a location reporting module. The location reporting module is a software program or function configured to monitor the location of at least one of the one or more objects represented by the respective node and to update spatial information associated with the at least one object of the respective node in the database system. For example, the nodes may represent objects that may be equipped with a GPS sensor or other means for repeatedly and automatically determining their location. The location information is automatically transferred from the location sensor to the location reporting module and stored by the location reporting module in the database system. This may ensure that any changes in the location of the device, for example the current location of a mobile robot, are automatically propagated to the location information stored in the database system. This allows a device that is moved towards a user with augmented reality glasses to automatically determine that it is in close proximity to a user wearing the AR glasses and whether data related to this object should be displayed via the AR glasses. For example, the location reporting module may be a software program that subscribes to a device-specific streaming service provided by the database system. Whenever each device changes its location, either actively or passively, sensors on the device and / or in the environment detect the device's change of location and its new location and update the device's location information in the database system accordingly. Updating the location information in the database triggers the sending of a message from the database system to the location reporting module of the container assigned to the node representing said device. The program logic of the augmented reality glasses is configured to use the updated location information of the device to check whether the distance between the user wearing the AR glasses and the device is small enough to display the data assigned to said device.

[0068] According to some embodiments, one or more of the nodes of the graph may comprise or be stored in association with two or more containers. Each of the containers may comprise software for monitoring and / or operating one or more physical objects. This may be beneficial because the topology of the graph and containers, as well as the selection of objects monitored and / or controlled by each container, may be flexibly adapted to the particular needs of a given industrial manufacturing facility.

[0069] The use of containers may have the advantage that the activities of a container do not have any hidden or unpredictable effects on the activities of other containers. It is possible to freely remove, add, or move a graph node representing a container in a graph without affecting the activities in other containers. Similarly, it is possible to change or replace software provided in a container without affecting software used to monitor and / or control other objects in an unexpected way. Furthermore, it is possible to move objects in the real world without the risk that the state or location of the object may interfere with software used to monitor and / or control physical objects represented by other containers. The changed location of the object is recognized by the location reporting module of the respective container, but this also does not affect the data and software stored in other containers. This can reduce downtime of the AR system, since individual nodes and respective hardware objects can be easily modified or replaced without the risk of disturbing the normal operation of other hardware objects and their software programs. Because containers are configured as standardized units of software, in other words, application packages including all their libraries and dependencies, containers can be quickly and reliably executed (i.e., run) regardless of the architecture of the database system.

[0070] In further embodiments, each of the containers advantageously automatically updates the configuration of the plant in response to movements and / or rearrangements of objects in the plant. For example, the configuration may be a file, or a set of files, or a set of one or more data records in a database. The configuration may represent the entirety of the equipment of the plant (machines, tools, robots, etc.) that may be used to perform a production process, with an indication of the position of each piece of equipment. An embodiment of the invention may have the advantage that the configuration is always kept up to date fully automatically. When an operator transports a particular tool, machine or machine part from any production unit to the next production unit, the configuration of the plant is automatically updated, since the positioning system automatically determines the rearrangement of the tool, machine or machine part and automatically updates the configuration of the plant.

[0071] If the above-mentioned node is to represent multiple physical objects including an object having a GPS sensor and further objects without a positioning sensor, even if these further physical objects do not have positioning sensors, the position information of the object having the GBS sensor can be used to at least approximately monitor the position of other physically associated objects.

[0072] According to an embodiment, the spatial information associated with each node's object includes one or more spatial coordinates of the object, the one or more spatial coordinates corresponding to one or more points on and / or inside the object's boundary surface.

[0073] As used herein, an "object boundary surface" is the surface of the smallest suitable 3D wrapper object that completely encloses the entire object. Typically, the 3D wrapper object is a rectangular box. Using the object boundary surface instead of the actual surface, and usually a much more topologically complex surface, to perform spatial operations in a database system (such as distance determination) can have the advantage that the speed of the spatial operations can be significantly increased. Due to the limited complexity of the wrapper object, especially a cube or rectangular box, geometric operations such as distance determination, calculation of intersection area, intersection volume, bond area, bond volume, etc. can be performed with much less CPU capacity than performing these operations on a "realistic" object surface representation.

[0074] A "boundary space" is a space proximate to a wearer of the AR glasses in which virtual objects may be displayed to the wearer. For example, a boundary space may or may not surround the user.

[0075] According to an embodiment, the bounded space determines whether a virtual object is rendered, whereby the virtual object is rendered if and only if it is within the bounded space. For example, rendering may include converting a 2D or 3D model of the object into a pixel representation of at least a portion of the object. Additionally or alternatively, it may include receiving data associated with the virtual object from one or more data sources and processing the received data such that at least a portion of the data is converted into the pixel representation of the data. Rendering is often a CPU-intensive process, and retrieving data from a database may involve retrieving the data over a network connection and thus generate significant network traffic.

[0076] A "boundary surface of a bounded space" is defined as a surface that defines (i.e., encloses) the bounded space.

[0077] In a further aspect, the present invention relates to a method of using an automation system for controlling an automated production process of an industrial plant, the industrial plant comprising a plurality of pieces of equipment for carrying out the production process, the plurality of pieces of equipment being spatially distributed within the industrial plant. The method comprises: An automation system, the automation system comprising an augmented reality system, the augmented reality system comprising: o a database system having stored data and spatial coordinates associated with each device of the plurality of devices; augmented reality glasses configured to display the data; o executable program logic coupled to the augmented reality glasses; and providing, receiving, by executable program logic, from a positioning system, an up-to-date spatial position of a user wearing the augmented reality glasses; determining, by a database system, data on devices located proximate to the user's most recent spatial location; receiving, by executable program logic, determined data of the device from a database system; controlling, by executable program logic, the augmented reality glasses to display at least a portion of the received data, where the displayed data allows a user to control and / or maintain an automated production process; Includes.

[0078] The following embodiments of the invention are described in detail, by way of example only, with reference to the drawings in which: [Brief description of the drawings]

[0079] [Figure 1A] FIG. 1 illustrates an automation system for controlling an automated production process in an industrial plant, according to one embodiment of the present invention. [Figure 1B] FIG. 2 is a top view of AR glasses worn on a user's head, according to one embodiment of the present invention. [Figure 1C] FIG. 2 is a top view of AR glasses worn on a user's head according to another embodiment of the present invention. [Figure 1D] FIG. 13 is a top view of AR glasses worn on a user's head according to yet another embodiment of the present invention. [Figure 1E] FIG. 1 illustrates a boundary space surrounding a user wearing AR glasses. [Figure 1F] FIG. 1 illustrates a bounded space that is adjacent to, but does not surround, a user wearing AR glasses. [Figure 1G] FIG. 2 illustrates two different bounding spaces for two different types of virtual objects. [Figure 1H] 1 illustrates a boundary space, the position and / or orientation of which is dynamically adapted to the line of sight of a user wearing the AR glasses. [Diagram 2] FIG. 2 illustrates an exemplary task instruction, according to one embodiment of the present invention. [Diagram 3] FIG. 2 is a diagram illustrating example nodes of a graph database, where the nodes represent task instructions. [Figure 4] FIG. 1 shows a flow chart of a method for operating an automation system. [Diagram 5] FIG. 2 illustrates an exemplary GUI displayed to an “operator” user. [Figure 6] FIG. 13 is a diagram illustrating an example of a GUI displayed to a "maintenance worker" user. [Figure 7] FIG. 1 illustrates the use of spatial anchors and shows an avatar of a remote user. [Figure 8] A diagram illustrating the coordinate system used to represent real-world devices and local user avatars in the virtual metaverse to remote operators. [Figure 9] FIG. 13 illustrates a flowchart of a method for operating an automation system according to another embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0080] The description of various embodiments of the present invention is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed in this specification.

[0081] An "industrial plant" or "plant" may refer, without limitation, to the industrial purpose of manufacturing, producing or processing one or more industrial products, i.e., the technological infrastructure used in the manufacturing or production process or processing carried out by the industrial plant. The industrial product may be any physical product, e.g., chemicals, biological products, pharmaceuticals, food, beverages, textiles, metals, plastics, semiconductors, etc. Additionally or alternatively, the industrial product may be a service product, e.g., recovery or waste processing, such as recycling, chemical processing, such as disintegration or dissolution into one or more chemical products. Thus, the industrial plant may be one or more of a chemical plant, a process plant, a pharmaceutical plant, a fossil fuel processing facility, such as an oil and / or natural gas well, a refinery, a petrochemical plant, a cracking plant, etc. The industrial plant may be either a distillery, a processing plant, or a recycling plant. The industrial plant may be any of the given above examples or a like combination thereof.

[0082] The platform may comprise equipment or process units such as any one or more of: heat exchangers, towers such as fractionators, furnaces, reaction chambers, cracking units, storage tanks, extruders, pelletizers, precipitators, blenders, mixers, cutters, hardening tubes, vaporizers, filters, sieves, pipelines, stacks, filters, valves, actuators, mills, transformers, conveying systems, circuit breakers, machinery, e.g., heavy duty rotating equipment such as turbines, generators, grinders, compressors, industrial fans, pumps, conveying elements such as conveying systems, motors, and the like.

[0083] Additionally, industrial plants typically include a number of sensors and at least one control system for controlling at least one parameter related to a process or process parameter in the plant. Such control functions are typically performed by a control system or controller in response to at least one measurement signal from at least one of the sensors. The controller or control system of the plant may be implemented as a distributed control system ("DCS") and / or a programmable logic controller ("PLC").

[0084] Thus, at least some of the equipment or process units of the industrial plant may be monitored and / or controlled to produce one or more industrial products. The monitoring and / or control may be performed to optimize the production of the one or more products. The equipment or process units may be monitored and / or controlled via a controller, such as a DCS, in response to one or more signals from one or more sensors. Additionally, the plant may include at least one programmable logic controller ("PLC") for controlling a portion of the process. An industrial plant may typically include multiple sensors that may be distributed in the industrial plant for monitoring and / or control purposes. Such sensors may generate large amounts of data. The sensors may or may not be considered part of the equipment. Thus, production, such as chemical production and / or service production, may be a data-heavy environment. Thus, each industrial plant may generate large amounts of process-related data. Those skilled in the art will appreciate that an industrial plant may typically include equipment that may include different types of sensors. The sensors may be used to measure one or more process parameters and / or to measure equipment operating conditions or parameters related to the equipment or process units. For example, sensors may be used to measure process parameters such as flow rate in a pipeline, level in a tank, temperature of a furnace, chemical composition of a gas, some sensors may be used to measure vibration of a grinder, speed of a fan, opening of a valve, corrosion of a pipeline, voltage across a transformer, etc. The difference between these sensors may not be based only on the parameters they sense, but also on the sensing principle that each sensor uses. Some examples of sensors based on the parameters they sense may include temperature sensors, pressure sensors, radiation sensors such as optical sensors, flow sensors, vibration sensors, displacement sensors, and chemical sensors such as those for detecting certain substances such as gases. Examples of sensors that employ different sensing principles may be, for example, impedance sensors such as piezoelectric sensors, piezoresistive sensors, thermocouples, capacitance sensors and resistance sensors, etc.

[0085] An industrial plant may be part of a plurality of industrial plants. The term "multiple industrial plants" as used herein is a broad term and should be given its ordinary and customary meaning to those skilled in the art and should not be limited to a special or customized meaning. The term may specifically, but not exclusively, refer to a compound of at least two industrial plants having at least one common industrial purpose. Specifically, a plurality of industrial plants may comprise at least two, at least five, at least ten, or more industrial plants that are physically and / or chemically bonded. A plurality of industrial plants may be combined such that the industrial plants forming the plurality of industrial plants may share one or more of their value chains, extracts, and / or products. A plurality of industrial plants may also be referred to as a compound, a compound site, a "Verbund" or a "Verbund site". Furthermore, the value chain production of the plurality of industrial plants to the final product through various intermediate products may be decentralized at various locations such as various industrial plants, or may be integrated into a Verbund site, or a chemical park. Such a Verbund site or chemical park may be, or may comprise, one or more industrial plants, where products manufactured in at least one industrial plant may serve as feedstock for another industrial plant.

[0086] As used herein, "automation system" refers to a system for the automatic control of processes, such as, for example, chemical, oil refinery, paper and pulp mills, etc. An automation system may use a network to interconnect sensors, controllers, operator terminals, and actuators. It may be based on open or proprietary standards. Process automation may be used to efficiently and safely operate power plants and factories in various types of industries.

[0087] "Production process" refers to any industrial process that, when used on or applied to input materials, results in a product. The product may be, for example, a mechanical, electronic, and / or chemical product. Thus, a production process may be any manufacturing or processing process, or a combination of processes used to obtain a chemical product. A production process may also include packaging and / or stacking of the product.

[0088] The terms "manufacture," "produce," or "process" are used interchangeably in the context of a production process. These terms can encompass any type of application of an industrial process to input materials that results in one or more products.

[0089] A "chemical product" in this disclosure may refer to any industrial product, such as a chemical product, a pharmaceutical product, a nutritional product, a cosmetic product, or a biological product, or any combination thereof. A chemical product may be entirely composed of natural components or may at least partially include one or more synthetic components. Some non-limiting examples of chemical products are organic or inorganic compositions, monomers, polymers, foams, pesticides, herbicides, fertilizers, feed, nutritional products, precursors, pharmaceutical or therapeutic products, or any one or more of their components or active ingredients. In some cases, the chemical product may be a product that can be used by an end user or consumer, such as a cosmetic or pharmaceutical composition. The chemical product may also be a product that can be used to further manufacture one or more products, for example, the chemical product may be a synthetic foam that can be used to manufacture shoe soles, or a coating that can be used for automotive exteriors. The chemical product may be in any form, such as a solid, semi-solid, paste, liquid, emulsion, solution, pellet, granule, or powder.

[0090] A production process may be continuous, in a sequence of activities, or it may be a batch chemical production process, for example, when based on a catalyst that requires recovery. One main difference between these production types is in the frequency with which data is generated during production. For example, in a batch process, production data extends from the start of the production process to the last batch, across the different batches produced in that run. In a continuous setup, the data is continuous, even with possible variations in the operation of the production and / or possible downtime due to maintenance.

[0091] A "process parameter" may refer to any of the variables associated with a production process, e.g., any one or more of temperature, pressure, time, level, etc. A process parameter may be indicative of a current state of an equipment, equipment part, material, or manufactured product. According to some embodiments, one or more of the process parameters are indicative of an operating mode of an equipment or equipment part. A process parameter may be measured by a sensor. Other process parameters may be desired parameter values ​​or ranges of values, and may be set and / or dynamically modified by a user. According to examples, a process parameter may be a numerical value or a binary signal value measured during a production process, e.g., via one or more sensors. Preferably, the process parameter is received and stored in association with time information indicative of the time at which the process parameter was measured.

[0092] "Input material" or "raw material" may refer to at least one feedstock or unprocessed material used to produce a chemical product. An input material may be any organic or inorganic substance, or combination thereof. Thus, an input material may be a mixture or may contain multiple organic and / or inorganic components in any form. In some cases, an input material may be an intermediate processed material, for example, from an upstream processing facility or plant.

[0093] As used herein, a "spatial database system" is a software application designed to enable the definition, creation, querying, updating, and management of spatial databases.

[0094] As used herein, a "spatial database" is a database that supports database operations optimized to store and query data representing objects defined in a geometric space. Spatial database systems allow for the representation of simple geometric objects such as points, lines, and polygons, and optionally complex structures such as 3D objects, topological coverages, and / or linear networks. While databases are typically designed to manage a variety of numeric and character data types, spatial databases provide additional functionality for efficiently processing spatial data objects. Spatial data objects are sometimes referred to as "spatial primitives" or "simple geometric objects." The term "geometry" may be used to refer to individual spatial data objects as well as aggregations of spatial data objects.

[0095] As used herein, a "spatial database operation" is a database routine configured to analyze spatial information of one or more spatial data objects. Spatial database operations are typically speed-optimized to efficiently process spatial data objects by using spatial indexes created for multiple spatial data objects. For example, a spatial database may support one or more of the following spatial database operations: spatial measurements (e.g., calculating line lengths, areas of polygons, distances between spatial data objects), spatial functions (e.g., modifying existing spatial data objects to create new spatial data objects by providing minimum bounding rectangles around spatial data objects, intersecting spatial data objects, merging spatial data objects), spatial predicates (e.g., performing true / false queries on spatial relationships between spatial data objects to check whether a polygon overlaps or is within a maximum distance from another spatial data object), etc.

[0096] As used herein, a "streaming database system" is a database system accessible via a subscription-based streaming service. Data in a streaming database system is often split into two tiers. The raw inputs, often called "streams", are immutable, append-only sequences of events. They are meant to be a historical record of what happened and when. The second tier is built from looking at the streams and assembling summaries, often statistical, about the events. For example, one might count the number of times an event occurred each day during the last month, or take an average value per week over the year. The analyses are typically stored in tables that are often similar in structure and behavior to traditional relational databases. Indeed, according to an embodiment, the streaming service is connected to and / or based on a traditional database of these results. Using a streaming database may have the advantage that it may reduce the size of the data and save on storage costs. For example, parameter values ​​collected by equipment sensors every second are received via a streaming interface, resulting in averages calculated over the course of a day. Storing only statistically aggregated parameter values ​​of at least some of the process parameters makes long-term tracking technically feasible.

[0097] In a further beneficial aspect, the streaming database makes it easy for developers to consolidate parameter values ​​received through one or more streams, since the streams are directly accessible in the streaming database system, as opposed to traditional databases where sequences of INSERT or DELETE events are hidden in an internal journal or ledger. Developers can adjust how the streams of new parameter values ​​are statistically aggregated and stored in tabular form. Furthermore, the streaming database system supports subscription-based data retrieval to dynamically determine, for example, which devices are in spatial proximity to a user wearing the AR glasses, whereby the device and / or the user may change location. The user may subscribe to a service using his / her current location as an argument or condition. The service continuously and automatically determines whether a device is in spatial proximity to the user, and causes the executable program logic of the AR glasses to generate and display a GUI with device data selectively from devices in spatial proximity to the user. Each movement of the user wearing the AR glasses may automatically trigger an update of the subscribed streaming service with the user's new location information. For example, Oracle's Stream, Apache's Kafka, or IBM's "Stream" can be used as a streaming database system.

[0098] The term "graph database system" as used herein refers to a database system that uses a graph structure for semantic queries with nodes, edges, and properties to represent and store data. The key concept of a graph database system is the graph (or edge or relationship). A graph relates data items in a store (e.g., equipment and associated task data records, location information, configuration parameters, or measurement parameters) to a collection of nodes and edges, where the edges represent relationships between the nodes. This relationship allows data in the store to be directly linked and often retrieved in a single operation. A graph database holds the relationships between data as a priority. Query relationships are fast because they are persistently stored in the database. Relationships can be intuitively visualized using a graph database, which is useful for data that is highly interconnected. For example, according to an embodiment, a graph represents equipment (e.g., machines, tools, and / or robots), equipment components, and their subcomponents in multiple different hierarchies. When a user approaches a particular piece of equipment and is in spatial proximity to this equipment, the graph database system can be used to quickly retrieve not only the data of one single node representing this particular piece of equipment, but also the data of any equipment components or sub-components (direct child nodes or second, third, or nth child nodes) in a single query operation. Thus, when a user is in spatial proximity to a piece of equipment, the executable program logic may receive all relevant parameters of this equipment and its sub-components and display it immediately via the AR glasses or cache some of the retrieved data for later display without further queries to the database system. Using a graph database can be particularly advantageous for drill-down analysis to retrieve data related to a particular maintenance task of a machine and one or more of its components.

[0099] The expressions "augmented reality glasses" or "AR glasses" as used herein refer to wearable computer glasses that add information to or in parallel with what the wearer sees. The superimposition of information on the field of view can be achieved, for example, by optical head mounted displays (OHMDs) or embedded wireless glasses with an augmented reality overlay. These systems have the ability to reflect projected digital images and allow the user to see through them or to see favorably by them. Some models have processors with limited processing power and perform only basic tasks, such as acting as a front-end display for remote systems with large CPU capacity, as is the case with smart glasses that utilize cellular technology or Wi-Fi. Modern AR glasses are effectively wearable computers that can run self-contained mobile applications. According to a preferred embodiment, the AR glasses are hands-free and can communicate with one or more external software programs, e.g., the executable program logic of the AR system, via optical sensors combined with natural language voice commands and / or gesture recognition capabilities.

[0100] According to an embodiment, the AR glasses may be configured to collect information from internal or external sensors, for example to determine the wearer's position and / or orientation, and represent part of the positioning system of the AR system. According to an embodiment, the AR glasses support wireless technologies, such as Bluetooth and / or Wi-Fi for data exchange with other components of the AR system, and / or GPS for position determination. The AR glasses may comprise an audio output interface, configured to output, for example, the voice of another user and / or sounds, such as alarm messages generated by the AR system, for example by executable program logic.

[0101] The expression "positioning system" as used herein is a monolithic or distributed technical system configured to determine the position of an entity, for example a user or another physical object. For example, some AR glasses are equipped with GPS sensors or indoor position sensors, gyroscopes and / or cameras, which are configured to continuously obtain data that allows to determine the current position of the user wearing the AR glasses.

[0102] The expression "spatial anchor" as used herein is a software function that allows to position a virtual reality object, such as a hologram, a virtual GUI or a manual, at a defined distance relative to said anchor. The anchor is mapped to a real-world coordinate system to represent a bounded coordinate system. According to some embodiments, the anchor is created by or with an executable program logic interoperable with the AR glasses, for example an augmented reality application. The anchor is then stored in a database system to make the anchor accessible to other users, for example remote users. A virtual reality application configured to create a virtual reality for a remote user can be configured to read the anchor created by the user wearing the AR glasses and to create and display some virtual objects at a defined distance and orientation relative to the anchor. The augmented reality application of the user wearing the AR glasses may do the same. The spatial anchor thereby allows two users to share the same visual experience. For example, the virtual object displayed at a given distance and orientation of the anchor can be a virtual representation of a device or a device part, or a virtual representation of a GUI designed to allow the user to control the device. According to one example, the spatial anchor can be implemented using Microsoft's Azure technology.

[0103] As used herein, the term "avatar" is a graphical representation of a user or a user's character or persona. When viewed through AR glasses, it may be in two-dimensional or three-dimensional form.

[0104] The phrases "near" or "close" or "close to" a user (or other object / entity) can mean, for example, within a user-defined or predefined maximum distance from this user (or from another object / entity). For example, the maximum distance can be less than 200 m, or less than 10 m, or less than 5 m, or less than 2 m, or less than 1 m. The maximum distance may also be specified, in some cases, in the form of a 3D object, e.g., a hemisphere or cylinder, with defined major and / or minor axes, or in the form of a 2D object, e.g., a circle, ellipse or polygon, such that the distance thresholds for "near" and "not near" objects may differ with various dimensions in the real-world 3D coordinate system.

[0105] According to some embodiments, a "boundary space close to a user" is a boundary space that surrounds the user. According to some embodiments, a "boundary space close to a user" is a boundary space that does not surround the user, but is close to the user. The maximum distance of a boundary space that is "close" to a user may vary depending on the embodiment. According to some examples, a boundary space that is "close" to a user is a boundary space that is located less than 200 m, or less than 10 m, or less than 5 m, or less than 2 m, or less than 1 m from the user at its farthest point.

[0106] For example, in some examples, the bounded space may be a bounded space that includes the user and has a radius in the range of 2m to 200m. This may be advantageous for alarm messages that may be associated with machines located in different rooms or floors, for example. In some further examples, the maximum distance of a bounded space that is "close" to the user may be, for example, a bounded space whose furthest point from the user is less than 10m away, for example less than 5m away from the user. This maximum distance may be useful for virtual objects that comprise, for example, video and text.

[0107] A "computing unit" may comprise or be a processing means or computer processor, such as a microprocessor, microcontroller, etc., having one or more processing cores. In some cases, the computing unit may be at least partially part of the equipment, e.g., a process controller, such as a programmable logic controller ("PLC"), or a distributed control system ("DCS"), and / or at least partially a remote server. Thus, the computing unit may receive one or more input signals from one or more sensors operably connected to the equipment. If the computing unit is not part of the equipment, it may receive one or more input signals from the equipment. Alternatively, or additionally, the computing unit may control one or more actuators or switches operably coupled to the equipment. The one or more actuators or switches may be part of the equipment.

[0108] A "memory storage device" may refer to a device for storing information in the form of data on a suitable storage medium. Preferably, the memory storage device is a digital storage device suitable for storing information in digital form, which is machine-readable, e.g., digital data readable via a computer processor. The memory may be a temporary or non-temporary storage device of information. Thus, the memory storage device may be realized as a digital storage device readable by a computer processor. More preferably, the memory storage device on the digital memory storage device may also be operated via a computer processor. For example, any part of the data recorded in the digital memory storage device may be written and / or erased and / or overwritten, partially or entirely, with new data by the computer processor.

[0109] Thus, the computational unit may manipulate one or more parameters associated with the production process, for example by controlling any one or more of the actuators or switches, and / or end effector units, via manipulation of one or more equipment operating conditions, preferably in response to one or more signals retrieved from the equipment.

[0110] An "interface" may be any hardware and / or software component of at least a portion of an apparatus or of another computing unit to which an object identifier is provided. In some cases, an interface may connect to at least one network, for example to interface two hardware components and / or protocol layers in the network. For example, an interface may be an interface between an apparatus and a computing unit. In some cases, an apparatus may be communicatively coupled to a computing unit via a network. Thus, an interface may be or comprise a network interface. In some cases, an interface may be or comprise a connection interface.

[0111] A "network interface" refers to a device or group of one or more hardware and / or software components that enable an operable connection to a network.

[0112] A "connection interface" refers to a software and / or hardware interface for establishing communication, such as transferring or exchanging signals or data. The communication may be wired or wireless. The connection interface is preferably based on or supports one or more communication protocols. The communication protocol may be a wireless protocol, e.g., a short-range communication protocol, such as Bluetooth, or WiFi, or a long-range communication protocol, such as a cellular or mobile network, e.g., a second generation cellular network, i.e., ("2G"), 3G, 4G, Long Term Evolution ("LTE"), or 5G. Alternatively or additionally, the connection interface may be based on a proprietary short-range or long-range protocol. The connection interface may support any one or more standards and / or proprietary protocols. The connection interface and the network interface may be the same unit or different units.

[0113] A "network" as described herein may be any suitable type of data transmission medium, wired, wireless, or a combination thereof. A particular type of network is not limited to the scope or generality of the present teachings. Thus, a network may refer to any suitable interconnection between at least one communication endpoint and another communication endpoint. A network may comprise one or more distribution points, routers, or other types of communication hardware. The network interconnections may be formed by physical hard wiring, optical and / or radio frequency methods. A network may specifically be or include a physical network made entirely or partially by hard wiring, such as an optical fiber network, or a network made entirely or partially by conductive cables or a combination thereof. A network may comprise, at least in part, the Internet.

[0114] "Equipment" may refer to any one or more assets in an industrial plant. As a non-limiting example, equipment may refer to any one or more, or combinations of, computing units or controllers, such as programmable logic controllers ("PLC") or distributed control systems ("DCS"), sensors, actuators, end effector units, transport elements such as conveyor systems, heat exchangers, such as heaters, furnaces, cooling units, reactors, mixers, mirrors, choppers, compressors, slicers, extruders, dryers, sprayers, pressure or vacuum chambers, tubes, bins, silos, and any other type of equipment used directly or indirectly for or during production in an industrial plant. Preferably, equipment specifically refers to assets, equipment, or components that are directly or indirectly involved in the production process. More preferably, those assets, equipment, or components that may affect the performance of a chemical product. Equipment may be buffered or not buffered. Furthermore, equipment may include mixing or not mixing, separating or not separating. Some non-limiting examples of non-buffered equipment that does not mix are conveyor systems or belts, extruders, pelletizers, and heat exchangers. Some non-limiting examples of buffered equipment that does not mix are buffer silos, bins, etc. Some non-limiting examples of buffered equipment with mixing are silos with mixers, mixing vessels, cutting mills, double cone blenders, hardening tubes, etc. Some non-limiting examples of non-buffered equipment with mixing are static or dynamic mixers, etc. Some non-limiting examples of buffered equipment with separation are towers, separators, extractors, thin film vaporizers, filters, sieves, etc. The equipment may further be or include storage or packaging elements, such as octabin fills, drums, bags, tank trucks, etc.

[0115] "Equipment operating condition" refers to any characteristic or value that represents the state of the equipment, such as any one or more of the following: set point, controller output, production sequence, calibration status, any equipment related warnings, vibration measurements, speed, temperature, contamination values ​​such as filter differential pressure, maintenance dates, etc.

[0116] "Parameter" in this context refers to any relevant physical or chemical property and / or its measure, such as temperature, direction, position, amount, density, weight, color, moisture, speed, acceleration, rate of change, pressure, force, distance, pH, concentration, and composition. A parameter may also refer to the presence or absence of a particular property.

[0117] "Actuator" refers to any component that serves to directly or indirectly move and control a mechanism associated with equipment such as a machine. Actuators may be valves, motors, drives, etc. Actuators may be operable either electrically, hydraulically, pneumatically, or a combination thereof.

[0118] A "computer processor" refers to any logic circuitry configured to perform basic operations of a computer or system, and / or generally, a device configured to perform calculations or logical operations. In particular, the processing means or computer processor may be configured to process basic instructions that drive the computer or system. As an example, the processing means or computer processor may comprise at least one arithmetic logic unit ("ALU"), at least one floating point unit ("FPU"), such as a numeric coprocessor or a numeric coprocessor, a number of registers, in particular registers configured to supply operands to the ALU and store the results of the operations, and memories, such as L1 and L2 cache memories. In particular, the processing means or computer processor may be a multi-core processor. In particular, the processing means or computer processor may be or comprise a central processing unit ("CPU"). The processing means or computer processor may be a complex instruction set computing ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, or a processor implementing other instruction sets or a processor implementing a combination of instruction sets. The processing means may also be one or more special-purpose processing devices, such as an application specific integrated circuit ("ASIC"), a field programmable gate array ("FPGA"), a complex programmable logic device ("CPLD"), a digital signal processor ("DSP"), a network processor, etc. The methods, systems, and devices described herein may be implemented as software in a DSP, microcontroller, or any other side processor, or as hardware circuitry within an ASIC, CPLD, or FPGA. It should be understood that the term processing means or processor may also refer to one or more processing devices, such as a distributed system of processing devices located across multiple computer systems (e.g., cloud computing), and is not limited to a single device unless otherwise specified.

[0119] A "computer-readable data medium" or carrier includes any suitable data storage device or computer-readable memory on which is stored one or more sets of instructions (e.g., software) embodying any one or more of the methods or functions described herein. The instructions may also reside, completely or at least partially, within a main memory and / or within a processor during execution by a computing unit, a main memory, and a processing device, which may constitute a computer-readable storage medium. The instructions may further be transmitted or received over a network via a network interface device.

[0120] 1A illustrates an automation system 100 for controlling an automated production process in an industrial plant. The industrial plant includes a plurality of pieces of equipment 122, 124 for performing the production process. In one embodiment, the plurality of pieces of equipment are spatially distributed within the industrial plant.

[0121] The automation system includes an augmented reality system including a database system 102 configured to store data and spatial coordinates associated with each of a plurality of devices, augmented reality (AR) glasses 104 configured to display the data, and executable program logic 106 coupled to the augmented reality glasses 104. For example, the AR glasses can be Microsoft's HoloLens, or an iOS-based device supporting, for example, ARKit, or an Android-based device supporting ARCore or a similar library. The executable program logic 106 can be integrated into the AR glasses or can be at least partially external to the glasses and operably coupled to the AR glasses via a wireless data exchange interface. The program logic may be used to generate a GUI and / or render digital geometric objects for display through the AR glasses. Additionally, the executable program logic may include interfaces for exchanging data with the database system 102 and the positioning system 108. In one embodiment, the augmented reality system includes the AR glasses 104 and program logic 106, and the database system 102 is either a component of the distributed control system 132 or a self-contained object or component of the plant communicatively coupled to the augmented reality system and / or the distributed control system 132. In one embodiment, the augmented reality system also includes a network 107 for communicatively coupling the AR glasses 104 to the database system 102. The augmented reality glasses 104 are associated with a human operator (i.e., an AR user) and include components such as electronics, optics (e.g., display, lenses, light source), power, local memory, processing unit, and executable software such as an AR application known to those skilled in the art, enabling the AR glasses to provide an enhanced (i.e., augmented) view of the plant environment when viewed through the glasses by a user.The AR glasses may also include an audio output interface, such as an integrated speaker and / or microphone, to enable a user wearing the glasses to communicate via voice with one or more remote users.

[0122] In one embodiment, as a user views the AR glasses display at different objects in the plant, such as equipment, sensors, actuators, different parts of the plant framework (e.g., floors, ceilings, walls, scaffolding, storage areas, etc.), the AR glasses 104 provide an overlay of data, also referred to as data AR objects or "virtual objects," on the AR glasses display that is representative of the data, such as text, photos, videos, holograms, GUIs, technical handbooks, and / or graphs. In one embodiment, the type and content of the AR objects displayed depends on the characteristics of the user of the AR glasses, the user's location in the plant, and / or the user's line of sight, as described in more detail below.

[0123] In one embodiment, the executable program logic 106 is configured to receive the user's most recent spatial location from a positioning system, such as positioning system 108. In one embodiment, positioning system 108 is configured to sense the user's spatial location within the plant.

[0124] According to an embodiment, the positioning system 108 comprises one or more sensors for dynamically identifying the position and / or orientation of a user wearing the AR glasses. For example, the positioning system may comprise a GPS module or an indoor positioning system. Additionally or alternatively, the positioning system may comprise one or more cameras used to capture the user's hand and arm gestures while interacting with virtual objects, such as a GUI displayed through the AR glasses. According to an embodiment, the positioning system 108 is configured to generate and / or use a spatial mapping mesh that covers at least a portion of the plant. For example, the spatial mapping allows for anchoring AR objects on or near real surfaces of objects in the plant, such as surfaces of equipment, sensors, actuators, or other surfaces in the plant, including surfaces of the plant base, through the use of a spatial mapping mesh. An embodiment of the present invention includes the use of commercially available spatial mapping mesh positioning systems, some of which are commercially available as applications executed by the AR glasses 104 and / or the program logic 106. One such system is World Locking Tools, which is available as open source code.

[0125] Additionally or alternatively, embodiments of the present invention include a pre-built spatial map (i.e., a spatial mesh) that defines the surfaces of objects in the plant, each surface of the object having a number of mesh points on which a respective AR object can be placed or locked, and by associating the coordinates of the lock mesh points with AR coordinates in the potentially moving AR coordinate system of the AR glasses, the AR object appears to be locked in place relative to its position relative to the corresponding plant object. In one embodiment, one or more spatial surfaces of objects in the plant may be represented by a triangular mesh to which an AR object may be anchored. Thus, an AR object placed in a predetermined position relative to physical world features remains fixed relative to those features regardless of the user's movement or the direction of the user's view through the AR glasses. For example, Microsoft's Azure Spatial Anchors Technology may be used to develop mixed reality applications that support AR glasses from various manufacturers. Azure Spatial Anchors enables the development of mixed reality applications that specify precise points of interest, called spatial anchors, and recall those points of interest from one or more supported devices, so that multiple local and / or remote users may share the same virtual coordinate system and the virtual objects contained therein. According to one embodiment, the spatial mapping mesh includes or is connected to spatial anchors. At least some of the spatial anchors may be interconnected, e.g., forming one or more groups of interconnected spatial anchors having a predetermined relative distance from each other. In one embodiment, one or more spatial anchors connected to any particular spatial anchor are defined to be one or more nearest spatial anchors. In one embodiment, each spatial anchor relates the coordinate system of an augmented reality system displaying data (e.g., an AR object) to a coordinate system of the real world.In other words, the positioning system 108 can be configured to merge one or more spatial anchors as viewed through the AR glasses (i.e., coordinates of one or more spatial anchors in the augmented reality coordinate system associated with the AR glasses) with real-world spatial coordinates of the real-world object to which the spatial anchors belong, to be stored in the database system in the form of a spatial mapping mesh, in order to obtain a real-world spatial location of a user of the AR glasses within the plant.

[0126] In another embodiment, the positioning system 108 is an indoor positioning system. The AR glasses 104 and / or the program logic 106 are configured to receive signals from Bluetooth beacons distributed throughout the plant and determine a position based on a received signal strength indication (RSSI) of the received Bluetooth beacon signal. Alternatively, which is particularly advantageous in a plant environment with a large amount of obstacles (i.e., non-open floor layout) that may result in multipath effects, shadow fading, and signal jamming, the AR glasses 104 and / or the program logic 106 are configured to receive cellular network and Bluetooth beacon signals as described in "Indoor Positioning System With Cellular Network Assistance Based on Received Signal Strength Indication of Beacon," IEEE Access, pp. 6691-6703, Dec. 30, 2019, which is incorporated herein by reference. In this embodiment, the RSSI values ​​of the received Bluetooth beacon signal are signal processed (missing values ​​are replaced and outliers are filtered out) to generate a smooth signal over the range. Then, the cell in which the AR glasses are located is determined according to the four strongest RSSI values, and a weighted multi-point positioning algorithm with cellular network assistance is used to calculate the real-time location of the AR glasses (i.e., the location of the AR user).

[0127] In yet another embodiment, the positioning system 108 is a cellular positioning system based on the 5G mobile wireless standard. The 5G positioning system provides extremely high network reliability and high positioning accuracy. In a further embodiment, the positioning system is a GPS positioning system, and the AR glasses include a GPS receiver and corresponding software and / or a GPS application configured to determine the spatial position of the AR glasses.

[0128] The executable program logic 106 is further configured to receive, from the database system 102, data corresponding to equipment located proximate to the user's most recent spatial location and to control the augmented reality glasses 104 to display at least a portion of the received data. In one embodiment, the executable program logic 106 is configured to receive, from the database system 102, data corresponding to any objects of the plant that are located proximate to the user's most recent spatial location.

[0129] In one embodiment, the executable program logic 106 determines the coordinates of a boundary space proximate to the user wearing the AR glasses 104 upon receiving the latest spatial position of the user wearing the AR glasses 104 from the positioning system 108. According to an embodiment, the executable program logic 106 is configured to determine the coordinates of the boundary space proximate to the user based on one or more of the following boundary space parameters: the latest spatial position of the user, the user's line of sight, the number of devices proximate to the user or in the user's line of sight (e.g., density of devices proximate to the user and / or in the user's line of sight), the number and / or data type of virtual objects corresponding to devices proximate to the user and / or in the user's line of sight, the configuration of the walls, floors, and / or ceilings of the plant base proximate to the user, and one or more user roles signed by the user wearing the AR glasses 104. Thus, the boundary space is defined by coordinates that define the boundary surface of the boundary space determined by the program logic. The program logic may also be configured to interpolate between the determined coordinates to add more coordinates that define the boundary surface of the bounding space at higher resolution.

[0130] According to embodiments, a bounded space is a 3D bounded space located within (i.e., defined by) a bounded surface, or a 2D bounded space located within (i.e., defined by) a bounded "surface" that is in a plane. Thus, the term "bounded surface" is defined for purposes of describing embodiments of the present disclosure as a 2D surface defined by a set of coordinates that encloses a 3D bounded space, or a 1D "surface" defined by a set of coordinates that encloses a 2D bounded space (i.e., as an illustrative, non-limiting example, a continuous line such as a curve that defines the circumference of a circle). Furthermore, the term "bounded space" is defined for purposes of describing embodiments of the present disclosure as including 3D and 2D bounded spaces.

[0131] In one embodiment, if the user's most recent spatial location places the user outside the plant boundary or inside an area of ​​the plant designated as off-limits, such as to protect sensitive information or limit access to highly hazardous areas (e.g., high voltage, air contaminants, infrastructure repairs), the program logic determines that the boundary space does not exist or, alternatively, that the boundary space has a very limited length in a particular direction, such as, for example, a direction pointing toward a hazardous or high priority area. Thus, the program logic may be configured to determine the coordinates of the boundary space based on the most recent spatial location of the user wearing the AR glasses 104.

[0132] In another embodiment, the program logic may determine and use a line of sight between a user wearing the AR glasses and the virtual object to determine whether or not to display the virtual object. For example, the executable program logic may dynamically receive position information of one or more physical objects, such as equipment, materials, and / or products. The executable program logic may determine when one or more of these physical objects are located on the line of sight between the user and the virtual object to be displayed. In one embodiment, if the program logic determines that one or more physical objects in the user's line of sight completely or partially obstruct the user's view, the control logic does not display the hidden portion of the virtual object. If the virtual object is completely hidden by a physical object in the line of sight, the virtual object is not displayed. The ratio of the hidden portion of the virtual object can be defined, for example,

[0133] In another embodiment, the program logic may determine the coordinates of the boundary space based on the density of equipment in proximity to the user. For example, the density of equipment in proximity to the user (e.g., equipment surrounding and in proximity to the user) and / or the density of equipment in the user's view direction (e.g., equipment in proximity to the user and in the user's line of sight direction determined by the orientation of the AR glasses or the orientation of the eyes) can be defined as the number of equipment per unit volume of the three-dimensional boundary space and the number of equipment per unit area of ​​the two-dimensional boundary space. The equipment density value can be associated with a spatial location in the plant and can be stored, for example, in the database 130 of the database system 102 and can be searched by the program logic 106. In one embodiment, the boundary radius parameter is determined based on the density of the equipment. For example, the boundary radius parameter can be inversely proportional to the density of the equipment. Thus, when the spatial location of the user wearing the AR glasses corresponds to a higher density of equipment than the average density, the program logic will result in a boundary radius parameter that is lower than the average. Once the boundary radius is determined, the program logic, for example, uses the boundary radius parameter to define an initial spherical boundary surface (or an initial hemispherical boundary surface, where the flat surface of the hemisphere coincides with the plant floor). The program logic can then modify the initial spherical boundary surface (or the initial circular boundary surface, in the case of an embodiment having a 2D boundary space) having an initial radius equal to the boundary radius parameter based on other boundary space parameters, such as the user's most recent spatial position, and / or the orientation of the AR glasses, and / or the orientation of the user's eyes, as described above. That is, the program logic can modify the initial spherical boundary surface to form a 3D non-spherical boundary surface, the coordinates of which are further based on any of the other boundary space parameters, as described above and further below. By making the boundary radius parameter inversely proportional to the density of the equipment, the embodiments of the present disclosure limit the amount of information provided simultaneously to the user of the AR glasses, thereby not overwhelming the user with too much data and reserving system bandwidth for other tasks.

[0134] In another embodiment, the program logic may determine the coordinates of the bounding space based on the number and / or data type of virtual objects corresponding to devices in proximity to the user and / or in the user's view direction. For example, the program logic may determine the total number of virtual objects corresponding to devices in proximity to the user. In one embodiment, the program logic determines the total number of virtual objects according to the radius of a sphere centered on the user in which each device is located. Then, based on a predefined virtual object maximum parameter that may be set and / or adjusted by the operator, the program logic determines the maximum radius such that the difference between the total number of virtual objects and the virtual object maximum is minimized without the total number of virtual objects exceeding the virtual object maximum.

[0135] In another embodiment, given a spatial position of the user (e.g., the latest spatial position), the program logic determines the total number of virtual objects according to the distance from the user measured along the viewing direction of the user wearing the AR glasses, at which the respective devices corresponding to the total number of virtual objects are located. Then, based on a predefined line of sight virtual object maximum parameter, which may be set and / or adjusted by an operator, the program logic determines a maximum distance such that the difference between the total number of virtual objects and the line of sight virtual object maximum is minimized without the total number of virtual objects exceeding the line of sight virtual object maximum. In a further embodiment, the program logic determines a plurality of maximum distances, one for each different viewing direction of the user located at the given user space position, each maximum distance being determined based on using the line of sight virtual object maximum parameter, as described above. Then, the plurality of maximum distances along each of the plurality of lines of sight define a plurality of coordinates on a bounding surface, or in other words, define a bounding surface, and thus define a bounding space (i.e., a space enclosed by a bounding surface). The bounding surface may have any shape and depends only on the plurality of maximum distances determined along the plurality of lines of sight.

[0136] The data type of the virtual object can be, for example, any of video data, text data, and audio data. A further data type is alarm data. Alarm data is data that needs to be urgently recognized by the user wearing the AR glasses. Thus, alarm data can be represented by audio data, such as, for example, a beep or siren, text data, such as, for example, a flashing red text, or video data, such as, for example, a short looped video message, or any combination of two or more of audio data, text data, and video data. In one embodiment, the display of the virtual object composed of alarm data by the AR glasses does not require any action on the part of the user. Actions include, but are not limited to, the user "clicking" on the virtual object text box, which indicates that text, video, or audio data corresponding to the further virtual object is available for display. For the purpose of distinguishing audio data, text data, and video data used as alarm data from audio data, text data, and video data that are not used as alarm data, audio data, text data, and video data used as alarm data are also referred to as emergency audio data, emergency text data, and emergency video data for the purposes of this disclosure.

[0137] In one embodiment, the program logic determines the coordinates of the boundary space based on the data type of the virtual object corresponding to the device in proximity to the user. For example, a virtual object composed of alarm data may be pre-assigned (by an operator of the system) a higher priority for display compared to a virtual object composed of video, text, or audio data. A virtual object composed of video text or audio data may also be pre-assigned a display priority value depending on the type of data (e.g., in one embodiment, audio data has a higher display priority than text data which has a higher display priority than video data). In one embodiment, the total display priority value at any given line of sight distance from the user is defined as the total sum of the display priority values ​​of the corresponding virtual objects associated with the devices that are along the line of sight between the user and the given line of sight distance. The operator may also define a maximum display priority value. In one embodiment, the program logic determines a maximum distance along the user's line of sight such that the difference between the total display priority value and the maximum display priority value is minimized without the total display priority value exceeding the maximum display priority value. In a further embodiment, the program logic determines multiple maximum distances based on the display priority values, one for each different line of sight of the user located at a given user space location. The maximum distances based on the display priority values ​​along the respective lines of sight define coordinates on a bounding surface, or in other words, define a bounding surface.

[0138] In another embodiment, the boundary space is one of two or more data type specific boundary spaces. As described above, the virtual object comprises an object of a data type selected from the group consisting of video, image, text, audio file, and alarm message. The executable program logic 106 is configured to dynamically determine the coordinates of the data type specific boundary space based on the data type of the virtual object displayed to the user wearing the AR glasses. For example, the user may select the data type specific boundary space to be used by the system via an audio command to the AR glasses, via a selection of a command displayed to the user by the AR glasses upon initial setup of the AR glasses, or via a selection by an operator of the system. That is, if a particular boundary space is selected to be (or be identified as) an alarm message specific boundary space (also referred to as an alarm boundary space), the program logic determines the coordinates of the boundary space based on the constraint that only virtual objects configured with alarm messages are displayed to the wearer (i.e., the user) of the AR glasses. Thus, the program logic is configured to determine a maximum line of sight distance for multiple lines of sight based on the user's most recent spatial position to define a boundary space and / or a radius of the boundary space around and in proximity to the user based solely on the boundary space including the virtual object configured as the alarm message, as described above.

[0139] Advantageously, to display more alarm messages than would be displayed if the bounding space were not identified as a data-specific bounding space, the program logic provides an alarm message-specific bounding space having a size and / or shape that can encompass a larger amount (or area). In other words, the program logic provides a data-specific bounding space having a size and / or shape that is different from the size and / or shape of a non-data-specific bounding space (i.e., a bounding space that includes virtual objects of all data types). In this manner, the size and / or shape of the bounding space is adjusted to be data-specific, thereby providing a bounding space that is "optimized" for viewing virtual objects of a particular data type, or in other embodiments, for viewing virtual objects comprised of any combination of one or more data types. In the sense of this disclosure, "optimization" according to the described embodiments means that the size and / or shape of the bounding space is determined to provide the user with the most meaningful and / or relevant data that the user should perceive and that can be assimilated by the user without overwhelming the user's cognitive ability to visually and / or audibly process information. The terms "size and / or shape" and "coordinates" of the bounding space used to describe the disclosed embodiments throughout this application are defined to have the same meaning. That is, once the coordinates of the bounded space are determined, which are the same as the coordinates of the bounding surface that encloses the bounded space, the size and / or shape of the bounded space is also determined.

[0140] In another embodiment, the program logic determines the coordinates of the bounded space based on the configuration of the walls, floors, and / or ceilings of the plant base proximate to the user. For example, the program logic is configured to determine a coordinate corresponding to the height of the bounded space based on the height of the room or area relative to the spatial location of the user. If the user is in an enclosed or semi-enclosed space such as a room, the program logic may also be configured to determine one or more lateral dimensions (i.e., dimensions lateral to height) of the bounded space. In one embodiment, the program logic determines the height of a room, compartment, or area proximate to and / or completely or partially surrounding the user by receiving measurement data from one or more sensors worn by the user. Alternatively or additionally, the program logic determines the height of a room, compartment, or area proximate to and / or completely or partially surrounding the user depending on a room ID, compartment ID, or area ID detected by a sensor worn by the user. For example, the rooms, compartments, and / or areas of the plant, such as walls and / or ceilings, as well as the equipment in some embodiments, may have electronically readable (e.g., scannable) ID tags. The AR glasses may include a scanner, such as a QR code enabled scanner, and the program logic may store an assignment table that assigns heights to room IDs and / or compartment IDs and / or area IDs.

[0141] In another embodiment, the program logic determines the coordinates of the boundary space based on the configuration of the walls, floors, and / or ceilings of the plant base proximate to the user in combination with one or more user roles assigned to the user. For example, the executable program logic is configured to receive user credentials from a user wearing the glasses and authenticate the user associated with a user ID. The user may be a new user requiring authentication or the user may already have a registered user account in a user database, for example including a user ID. The user ID may be associated with one or more user roles, for example an "operator" role, or a "maintenance" role. Additionally or alternatively, each user role may optionally include one or more user privileges, as described above.

[0142] According to one embodiment, the program logic is configured to receive one or more user roles associated with the user ID. For example, the user role can be an "operator", i.e., a role of a user who is responsible for controlling and / or monitoring one or more steps of a production workflow, or a "maintenance worker", i.e., a role of a user who is responsible for performing maintenance terms to ensure that at least one piece of equipment operates properly. Based on the role(s) assigned to the user ID, the program logic is configured to determine the extent of the boundary space, such as determining whether the boundary space includes two or more rooms or two or more floors of the plant, and / or the extent of the boundary space proximate to the user at any spatial location in the plant. For example, if the user wearing the AR glasses has a maintenance role rather than an operator role, the user may be presented with virtual objects associated with equipment in multiple different rooms and / or floors of the industrial plant in relation to the user's latest spatial location. Because maintenance workers are typically responsible for technically complex tasks that may require detailed technical knowledge of the equipment that is not typically available or even necessary for a user with an “operator” role, a user with a maintenance role may have a bounded space of much larger size and / or shape to encompass virtual objects associated with a variety of equipment, even equipment that is not located in close proximity to the user's current spatial location, such as equipment located in different rooms or on different floors of the plant.

[0143] The scope of the present disclosure also includes embodiments in which the program logic uses one or more of the above-mentioned boundary space parameters, such as the user's most recent spatial location, the orientation of the user's AR glasses, the orientation of the user's eyes, the user's line of sight, the number of devices proximate to the user or within the user's line of sight (e.g., the density of devices proximate to the user and / or the density of devices in the user's line of sight), the number and / or data type of virtual objects corresponding to devices proximate to the user and / or within the user's line of sight, the configuration of the walls, floors, and / or ceilings of the plant base proximate to the user, and one or more user roles assigned to the user wearing the AR glasses 104, in any combination, in determining the coordinates of the boundary space proximate to the user wearing the AR glasses (i.e., the coordinates that define the boundary surface surrounding the boundary space).

[0144] In another embodiment, the program logic determines whether the boundary space determined by any of the above-mentioned embodiments includes one or more instruments that are not entirely within the boundary space. For example, when the program logic determines a boundary space defined by a boundary surface that surrounds the boundary space, the program logic determines whether any instrument that has at least one corresponding virtual object located within the boundary space has at least one coordinate that is located outside the boundary space. An instrument that is defined by two or more coordinates, for example by coordinates that define a corner of the instrument, may be bisected by the boundary surface such that some of the coordinates corresponding to the instrument are within the boundary space and other parts of the coordinates corresponding to the instrument are outside the boundary space. In one embodiment, the program logic adjusts the boundary surface that bisects the instrument so that the instrument is entirely within the boundary space, and therefore also adjusts the display to include virtual objects that are included within the newly adjusted boundary space. In another embodiment, the program logic adjusts the boundary surface that bisects the instrument so that the instrument is entirely outside the boundary space, and therefore also adjusts the display to include only virtual objects that are included within the newly adjusted boundary space.

[0145] Once the program logic 106 determines coordinates of a boundary space that is proximate to a user wearing the AR glasses 104 or has at least a portion that is not proximate to a user wearing the AR glasses 104, which may occur when the program logic uses one or more user roles to determine the coordinates of the boundary space, the program logic 106 displays one or more virtual objects to the wearer of the glasses via the AR glasses 104 if and only if the coordinates of the virtual objects are within the boundary space. The displayed virtual objects include data such as text, video, images, audio, or alarm data. The data is associated with monitoring and / or control of an automated production process.

[0146] For example, in one embodiment, the program logic transmits the latest spatial location of the user wearing the AR glasses and the coordinates of the boundary space (e.g., the coordinates of the boundary surface surrounding the boundary space) to the database system 102, and receives from the database system the virtual objects having coordinates contained within the boundary space defined by the coordinates of the boundary space. Thus, the user (e.g., the wearer) of the AR glasses sees an overlay of the virtual objects contained within the boundary space with the real-world objects of the plant. The user can then use the text, image, video, audio, and / or alarm data corresponding to the displayed virtual objects to maintain and / or control the respective equipment to which the virtual objects correspond, and / or obtain operating status, parameter settings, and / or educate himself / herself on how the equipment operates and / or on the role of different equipment in the production process.

[0147] Furthermore, the program logic is configured to continuously update the coordinates of the boundary space and the coordinates of the virtual objects displayed in the boundary space according to the latest spatial position of the user wearing the AR glasses. Furthermore, the coordinates of the boundary space are determined according to the orientation of the AR glasses and / or the orientation (line of sight) of the eyes of the user wearing the AR glasses. Thus, when a user wearing or operating the AR glasses moves to a different spatial position and / or when the user is stationary with respect to the spatial position but changes the line of sight through the glasses by changing the orientation of the AR glasses (e.g., a user wearing the AR glasses as a headpiece moves the headpiece by the movement of the user's head), the program logic determines a new boundary space. The program logic 106 then displays one or more virtual objects to the wearer of the glasses via the AR glasses 104 if and only if the coordinates of the virtual objects are within the new boundary space. The virtual objects displayed in the new (i.e., updated) boundary space may include some of the virtual objects displayed in the previous boundary space depending on the overlap between the new boundary space and the previous boundary space.

[0148] In another embodiment, the program logic is configured to continuously update the coordinates of the boundary space and the coordinates of the virtual object displayed in the boundary space according to the latest spatial position of the user wearing the AR glasses. According to some embodiments, data indicative of the orientation of the AR glasses is also used to determine and update the boundary space. For example, the AR glasses may include a gyroscope and / or an accelerometer for determining the orientation of the AR glasses. Additionally or alternatively, the executable program logic determines and updates the coordinates of the boundary space according to the orientation of the eyes of the user wearing the AR glasses. Several methods and respective sensors for determining the orientation of the user's eyes ("gaze") are known and can be used in embodiments of the present invention. For example, an overview of existing eye tracking systems is provided in "Eye Tracking and Head Movement Detection: A State-of-Art Survey" by Al-Rahayfeh A, Faezipour M et al., published on November 6, 2013 in IEEE J Transl Eng Health Med, Volume 1, Series No. 2100212. doi:10.1109 / JTEHM.2013.2289879.

[0149] 1B-1H show top views of the AR glasses 104 worn on a user's head 105. The user looks through the AR glasses with the user's eyes 107. The eye orientation, i.e. the direction in which the user is looking, in other words the direction of the user's gaze, is defined by an eye orientation direction 109 perpendicular to the eye pupil 111. The AR glasses have an orientation direction 113, which is defined by the orientation of the user's head 105. This application uses the terms "user's gaze" and "direction of view" interchangeably. The direction of view may be determined, for example, by eye tracking techniques, for example, by detecting the "orientation" of the AR glasses and / or by detecting the eye orientation.

[0150] For example, in embodiments where the AR glasses do not include sensors for detecting the orientation of the user's eyes, the orientation of the AR glasses may be used as an indicator and / or predictor of the user's view direction. According to other embodiments, the AR glasses, or other components of the automated system external to the AR glasses, may include one or more sensors, e.g., cameras, for detecting the orientation of the user's eyes. In this case, the orientation of the user's eyes detected by these sensors is transmitted to executable program logic and used to determine the coordinates of the bounded space.

[0151] FIG 1B shows an orientation 113 of the AR glasses pointing in the same direction as the eye orientation 109 of a user wearing the AR glasses. FIG 1C and FIG 1D show the orientation 113 of the AR glasses and the eye orientation 109 of a user wearing the AR glasses pointing in different directions. For example, FIG 1C represents the orientation 113 that should be directly in front of the AR glasses while the user is looking to the right (i.e., through the right part of the glasses) and FIG 1D represents the orientation 113 that should be directly in front of the AR glasses while the user is looking to the left (i.e., through the left part of the glasses).

[0152] In one embodiment, the positioning system 108 includes an optical sensor system including a camera or other optical device, such as a laser, that measures the position of the pupil in the eye to determine the eye orientation (also referred to as eye orientation direction) of a user wearing the AR glasses. As mentioned above, the orientation of the AR glasses worn by a user is not necessarily the same as the orientation of the user's eyes, and the coordinates of the boundary space, in some embodiments, depend not only on the spatial position of the user wearing the AR glasses, but also on the orientation of the AR glasses (i.e., the orientation of the user's head) and / or the orientation of the user's eyes (i.e., the direction in which the user is looking through the glasses).

[0153] FIG. 1E illustrates a boundary space surrounding a user wearing the AR glasses 104 according to one embodiment. The boundary space may be a 3D space, with cross-sections of the boundary space surface shown in dotted lines. Depending on the embodiment, the boundary shape may have different forms, such as, for example, a sphere, a hemisphere, a polygon-based 3D body, a cube, a cuboid, etc. The executable program logic 106 is configured to display a virtual object to the user through the AR glasses 104 only if the virtual object is located within the boundary space 150. For example, the virtual object 115 located within the boundary space 150 is displayed, and the virtual object 117 is not displayed through the AR glasses. The object 117 is displayed only if the user approaches the position of the virtual object 117 in the augmented reality coordinate system (or the object approaches the user) such that the virtual object 117 is within the boundary space 150. The boundary space may be dynamically updated according to the current position of the user wearing the glasses, such that the user is always in a predetermined position inside or outside the boundary space and relative to the boundary space. For example, updates may be implemented so that the user is always in the center of the bounded space, a predetermined distance "behind" the bounded space, a predetermined point within a portion of the bounded space towards the user's rear, or another predetermined location.

[0154] 1F illustrates a boundary space 150 that is adjacent to, but does not surround, a user wearing AR glasses, according to another embodiment. In the illustrated embodiment, the boundary space has a rectangular parallelepiped shape. In other respects, the boundary space may have characteristics, for example, as described for the embodiment illustrated in FIG. 1E.

[0155] FIG. 1G illustrates two different boundary spaces for two different types of virtual objects. For example, boundary space 150 may be a smaller volume than boundary space 152. Boundary space 150 may be completely contained within boundary space 152 or may overlap with it. The two boundary spaces 150, 152 may have the same or different forms. In the illustrated example, boundary space 152 is a trapezoid-based body, while boundary space 150 is a spheroid. In the illustrated example, the executable program logic uses boundary space 150 to determine whether a virtual object of type "video" is displayed through the AR glasses, i.e., if a virtual object of type "video" is placed within boundary space 150, it is displayed, as in the case of object 115. Otherwise, it is not displayed, as in the case of object 117. The executable program logic uses the boundary space 152 to determine whether a virtual object of type "alarm message" is displayed via the AR glasses (or emitted via the audio output interface in the case of an audio alarm), i.e., the alarm message 118 is displayed and / or output via the AR glasses because it is within the alarm type boundary space 152. The alarm 119 is not displayed / output because it is outside the boundary space 152.

[0156] This may be advantageous since a human user cannot normally view multiple videos in parallel. For example, by displaying only instructional videos on how to operate a particular machine close to the user, the user is prevented from being distracted and making mistakes while operating the machine. It also reduces the amount of data that needs to be sent to the AR glasses, avoiding the need for computational power to render and display multiple videos. In contrast, displaying alarm messages in a much larger space has the advantage that important status messages that may be vital to the manufacturing process are communicated to the user even if the user is not in the immediate vicinity of the machine. In some embodiments, the boundary space of the alarm message is quite large, and the user may even be in another room or on another floor. Alarm messages can usually be processed quickly by the user and only slightly tax the bandwidth and computational power of the AR system. Thus, by using different boundary spaces for different types of virtual objects (e.g., videos, text, alarm messages, etc.), the manufacturing process, or the maintenance of the associated machines, can be monitored to significantly improve safety and efficiency.

[0157] According to an embodiment, the size of the virtual object-specific boundary space is selected to ensure that the virtual object displayed within this boundary space can be perceived and interpreted by the human visual system, i.e., at the edge of the field of view, the eyes have low visual acuity. Therefore, displaying a complex and finely structured virtual object at the edge of the field of view, for example an object with a lot of text, may bring little benefit to the user and may even distract the user and increase the cost of CPU resources. By generating and using a suitable boundary space of the virtual object type by the executable program logic to determine whether a given virtual object should be displayed or not, operation errors are avoided.

[0158] FIG. 1H shows a boundary space, the position and / or orientation of which is dynamically adapted to the line of sight of a user wearing the AR glasses. In the illustrated embodiment, the AR glasses are equipped with a sensor for detecting the user's line of sight, independent of the orientation of the AR glasses. In this case, the position and / or orientation of the boundary space (or multiple boundary spaces, if applicable) may be dynamically updated to follow and reflect changes in the direction of the user's line of sight. For example, the update may include repositioning the boundary space so that the center of the boundary space is always on a point on the user's line of sight (here line of sight). Thus, as shown in FIG. 1G, before the user moves his eyes to the right, virtual object 115 is shown, but 117 is not shown. As shown in FIG. 1H, after the user moves his eyes to the right, virtual object 115 is shown, but 117 is not shown.

[0159] According to other embodiments, the position and / or orientation of the boundary space (or multiple boundary spaces, if applicable) is dynamically updated to track and reflect changes in the orientation of the AR glasses worn by the user and / or changes in the user's line of sight (eye direction).

[0160] According to yet another embodiment, the database system 102 is a graph database system, a spatial database system, and / or a streaming database system. The database system 102 also includes a database 110.

[0161] In one embodiment, the database system 102 is a spatial database system and the database 110 is a spatial database configured to store respective data based on a spatial coordinate associated with the respective data, and each data is searched and retrieved based on the spatial coordinate associated with the respective data. For example, the spatial database system 102 of the present invention stores a plurality of spatial coordinates, each spatial coordinate including corresponding data related to an object in the plant located at that spatial coordinate. For example, given a particular piece of equipment in the plant, e.g., a pump, the spatial database stores one or more spatial coordinates of the pump, an optional description of each spatial coordinate (e.g., spatial coordinates 1-4 represent the four corners of the pump on the floor of the plant, spatial coordinate 5 represents the high point of the pump, etc.), and data associated with the pump, e.g., task instructions for the pump. As an example embodiment, the task instructions (for a pump) may include, without limitation, one or more of a training video, one or more photographs and / or training photographs, associated assets (e.g., other equipment or plant objects that may house or work in conjunction with the pump, such as equipment that supplies raw materials to the pump or receives products or pump outflow), short task instruction text, detailed task instruction text, and a list of task instruction categories. Task instructions are further described below in connection with FIG. 2. In particular, the database system 102 may be a graph database system, i.e., a spatial graph database, that supports rapid geometric operations that support rapid drill-down analysis of equipment-related data, including rapid determination of distances to other objects and / or users.

[0162] In one embodiment, the executable program logic 106 is configured to perform a query of the spatial database 110 based on the latest spatial location to retrieve data from the spatial database 110 for equipment that is in proximity to the user's latest spatial location. The query can be, for example, a traditional SQL query that is repeated multiple times using the user's current location as an argument, or it can be a subscription to a streaming service that is updated as the user changes location. In one embodiment, the proximity of the locations is defined by a user-adjustable proximity parameter of the executable program logic 106. The user may set the proximity parameter to any radius measured from the user's spatial location, or to any radius that defines a hemisphere with a flat surface located on the plant floor and centered on the user's latest spatial location.

[0163] In another embodiment, the executable program logic 106 is configured to perform a query of the spatial database 110 to retrieve data from the spatial database 110 for equipment located proximate to the latest spatial location and located within the user's field of view of the AR glasses based on the user's most recent spatial location and the user's field of view through the AR glasses. For example, the proximity of the location is defined by a user-adjustable proximity parameter of the executable program logic 106. The user may set the proximity parameter to any radius measured from the user's most recent spatial location or any radius located on the plant floor and defining a hemisphere with a flat surface centered on the user's spatial location. In response to the query, the executable program logic 106 receives data for equipment located within a 3D cone defined by the current field of view of the AR glasses and having a length defined by a user-defined radius.

[0164] In one embodiment, data of equipment located in proximity to the user's most recent spatial location and optionally within the field of view of the user of the AR glasses is received by the executable program logic even if such data is obstructed by obstacles in the plant from the view of the user of the AR glasses. Thus, if the second equipment is partially or completely hidden behind the first equipment, the executable program logic 106 is configured to control the augmented reality glasses 104 for display of data associated with the second equipment by placing the data associated with the second equipment (i.e., the second object) in the vicinity of the second equipment such that the first equipment does not obstruct the view. In one embodiment, the data associated with the second equipment is displayed with a smaller font and / or a smaller image and / or a pointer pointing in the direction of the hidden or partially hidden second equipment to indicate that the data is associated with an equipment at least partially hidden from the view of the user of the AR glasses 104.

[0165] In another embodiment, the spatial database 110 of the spatial database system 102 is accessible via a subscription-based streaming service. The executable program logic 106 is configured to receive data of devices located in proximity to the latest spatial location of the user of the AR glasses from the spatial database 110 via the streaming service. That is, as opposed to retrieving data of devices located in proximity to the latest spatial location of the user from the spatial database 110 via a query of the spatial database 110, as described above, the executable program logic 106 and the spatial database 110 are configured, according to this embodiment, such that the executable program logic 106 receives data of devices located in proximity to the user's latest spatial location based on the latest spatial location of the user in communication with the streaming service provider via the service of any commercially available third-party streaming service. The spatial database system 102 including the spatial database 110 accessible via any subscription-based streaming service may also be referred to as a streaming spatial database system.

[0166] According to some embodiments, the database system 102 is a graph database system, and the database 110 is a graph database supporting fast geometric operations. Thus, the database system can be considered as a combined graph database and spatial database. The graph database is configured such that nodes represent objects in the physical world, in particular equipment and equipment components or sub-components, and edges represent the real-world spatial and organizational relationships of these equipment, components and sub-components. Equipment and component related data, such as manuals or GUIs for setting process parameters or location information of said equipment or components, may be stored, searched and retrieved based on spatial coordinates associated with the respective data, and the executable program logic 106 is configured to perform a query of the graph database based on the latest spatial location and, optionally, the field of view of the AR glasses 104, to search from the graph database for data of equipment located in the vicinity of the user's latest spatial location and located within a field of view (e.g., within a field of view 3D cone) defined with reference to a 3D cone having a vertex based on the AR glasses with a longitudinal symmetry axis pointing in the field of view direction of the user of the AR glasses.

[0167] According to some embodiments, the graph database 110 of the graph database system 102 is accessible via a subscription-based streaming service, as described above in connection with the spatial database. In this embodiment, the executable program logic 106 is configured to receive, via the streaming service, from the graph database 110 data of devices located in proximity to the most recent spatial location of the user of the AR glasses 104. A graph database system 102 including a graph database 110 accessible via any subscription-based streaming service may also be referred to as a streaming graph database system.

[0168] Graph databases (and streaming graph databases) can be configured such that objects of a plant are stored as interconnected nodes of a graph, with each node including one or more spatial coordinates of an object in the plant and data corresponding to the object. Given the graph database architecture, the graph databases (and streaming graph databases) of the present invention enable fast lookup and access of data associated with a given object in the plant along the topology of the graph, thereby evaluating the spatial coordinates of a user of AR glasses within the plant.

[0169] In one embodiment, the objects in the plant include, without limitation, one or more sensors, of which sensors 112, 114, and 116 are exemplary embodiments, one or more actuators, of which actuators 118 and 120 are exemplary embodiments, at least one piece of equipment 122, 124, such as a machine (e.g., a robot or stationary manufacturing machine), a machine component or subcomponent, a tool, and / or a raw material 126 and 128. In some embodiments, one or more actuators and / or sensors, such as actuators 118, 120 and sensors 114, 116, are connected to the plant equipment 122, 124 to operate the equipment and / or components of the equipment to sense parameters associated with the equipment and / or with the raw materials 126, 128 being processed by the equipment. Parameters that may be sensed include, but are not limited to, pressure, temperature, humidity, and parameters associated with the movement of equipment (e.g., machines such as stationary machines or robots) or components of equipment, or of raw materials being or being processed by the equipment (e.g., conveyor speed, speed of fluid flow in pipes, rotation speed of mixers, movement of automated components, etc.). The scope of the present invention also includes sensors such as sensor 112 that are not connected to any plant equipment to measure ambient plant conditions, such as, for example, temperature inside the plant, ambient lighting, or particulate concentration in the air. The scope of the present invention also includes objects within the plant, such as equipment 122 and robot 124, that are not connected to any sensors or actuators. Plant equipment may also include inert (i.e., passive) structural components of the plant (i.e., plant components such as trellises, walls, ceilings, heaters, air conditioners, and support structures that are not involved in the processing of raw materials or intermediate materials, for example).

[0170] In another embodiment, the database system 102 optionally includes isolated virtual user spaces, so-called "containers" 130. The containers may be implemented, for example, as Docker containers. Each container 130 is associated with a respective node and configured to continuously determine current spatial information associated with the object of the respective node (e.g., object position and / or orientation) and immediately update position information of this object in the database 110. For example, each container may comprise software functionality interoperable with the positioning system 108 and configured to continuously receive current position information of the respective object. Additionally or alternatively, one or more of the containers may comprise software functionality interoperable with one or more positioning sensors attached to the object represented by the node associated with said container and may receive position information directly from the positioning sensors.

[0171] According to a preferred embodiment, the database 110 or another data storage repository comprises a configuration of the plant, specifying the type and location of a number of physical objects that make up the plant, e.g., of equipment, of equipment parts, and optionally of raw materials and / or products. The containers 130 in combination with the position sensors and / or positioning system 108 advantageously allow a fully automated update of the configuration of the plant in response to any movement and / or relocation of objects in the plant. For example, robots move around the plant, and components of the automated system may also move to different positions as part of their operation. When the containers are separated from each other, each container is a "standardized unit of software" and contains all the libraries and dependencies required by all the software programs that operate within the container, including the automatic positioning and position information update functions related to the respective objects of the plant, which allows the software in the container to be executed reliably regardless of the architecture of the database system and regardless of the state of other containers and their respective objects.

[0172] According to some embodiments, e.g., the embodiment shown in FIG. 1A, at least one piece of equipment comprises one or more robots. The one or more robots can be configured to perform one or more tasks in a manufacturing process, e.g., transport raw materials to a machine or cut or glue materials. Additionally or alternatively, one or more of the robots can be configured to move and operate as a physical representative of a remote user, e.g., human operator 702 shown in FIG. 7. The robot can comprise an interface for receiving control commands from a robot external robot control module, whereby the control commands determine the position, orientation, movement or task performed by the robot. According to some examples, the control commands are adapted to control the movement, position and / or orientation of the robot such that the movement, position and / or orientation of the robot corresponds to and reflects the movement, position and / or orientation of an avatar of the remote user in a coordinate system associated with the augmented reality glasses.

[0173] According to some examples, the robot representing the remote user and acting as a physical representation of the remote user comprises one or more sensors, such as a camera and a microphone, configured to capture sensor data (e.g., images, videos, captured audio signals) and transmit the sensor data to a virtual reality application 701 (see FIG. 7 ) to enable the virtual reality application to generate a virtual reality in response to the sensor data, such that a bidirectional real-time synchronization between the virtual reality presented to the remote user and the physical environment of the robot is achieved, i.e., the movement of the remote user (or at least the head of the remote user) triggers a corresponding movement of the robot, and the optical and acoustic signals acquired by the sensors of the robot are provided to the VR application and used by the VR application to create a virtual world and to display and / or acoustically output it to the remote user, whereby at least a part of the virtual world reflects the acquired acoustic and optical signals.

[0174] According to some examples, the robot includes a first interface for receiving control commands from a robot control module of the control system 132 and transmitting sensor data to the control system 132. This may have the advantage that the control system 132, which may be configured to coordinate multiple pieces of equipment during a manufacturing process, may use and control the robot to perform specific tasks in the manufacturing process.

[0175] According to some examples, the robot comprises a second interface for receiving control commands from the AR system 101, e.g., from an AR application 106 comprised in the AR system, and / or from a VR application 701 (e.g., as shown in FIG. 7). For example, the AR system 101 may also comprise a robot control module configured to receive sensor data from the robot and / or to send control commands to the robot for controlling the movement, orientation, position, and / or tasks performed by the robot. The current position and orientation of the robot are continuously reported and stored in the database 110. When the robot changes position and the updated coordinates of the robot are stored in the database, this event may trigger the display or hiding of a virtual object in the AR glasses of the local user. This may have the advantage that the robot is seamlessly integrated into the manufacturing process coordinated by the control system through the first interface and also integrated into the world of virtual objects created by the AR or VR application. In some examples, the robot may comprise a conflict resolution module for resolving conflicts that may be caused by incompatible control commands received through the first and second interfaces. For example, the conflict resolution module may resolve the conflict by overriding or ignoring the conflicting command provided by the remote user's VR application and rather executing the conflicting control command received via the first interface.

[0176] Just as the data in the database system represents a digital twin, in the sense that the database system 102 is configured to compute, or enable computation by the executable program logic 106, a complete replica of the plant and its components, the database system 102 is also considered to store location information and other data that describes the physical objects of the plant. The virtual replica of the plant may include all plant objects or, alternatively, may include plant objects that affect the production process and may include one or more real-world spatial coordinates associated with each plant object, as well as data corresponding to each plant object.

[0177] According to another embodiment, the spatial information associated with the object of each node includes one or more spatial coordinates of the object. For example, in a further embodiment, the one or more spatial coordinates correspond to one or more points on and / or inside the boundary surface of the object. The boundary surface of the object is defined to be the outer surface of a wrapper object that comprises the object. The spatial coordinate inside the boundary surface may be, for example, the center of mass of the object or the geometric center of the object. The wrapper object may be a rectangular 3D box, or a cube or sphere.

[0178] In another embodiment of the present invention, the data stored in the database system includes one or more task instructions associated with each device.

[0179] 2 illustrates an exemplary task instruction 200 (i.e., an exemplary AR object as viewed by a user through AR glasses) according to one embodiment of the present invention. The task instruction AR object 200 represents the task instruction "Add Start Line 1" (A2) that is part of the operator's view through the AR glasses shown in FIG.

[0180] The task instructions visualized (i.e., displayed) by the AR glasses and superimposed on the view of the plant seen by the user through the AR glasses include, but are not limited to, one or more of short text 202, detailed text 204, photo 206, and / or video icon 208. In one embodiment, the program logic 106 associates the spatial coordinates (defined with respect to the real-world coordinate system) stored in the database and linked to (i.e., corresponding to) the task instructions of the respective equipment with a spatial coordinate system (hereinafter "AR glasses coordinate system") used by the AR glasses to display virtual objects, and anchors the task instructions to the spatial coordinates defined with respect to said coordinate system. In one embodiment, the AR glasses coordinate system to which the task instructions are anchored is an anchor point of a mesh network covering (i.e., defining) the respective equipment, and the program logic 106 continuously updates the coordinates of the virtual objects in the AR glasses coordinate system so that the display of the task instructions does not move when the user of the AR glasses moves or changes the view direction.

[0181] In further embodiments, the content of the task instructions includes text, pictures, and videos that provide information about the respective equipment, such as operational parameters, including, for example, maximum and minimum values ​​of the operational parameters, maintenance requirements and / or history, raw materials used and products produced, pictures of the respective equipment to which the task instructions correspond, etc. In one embodiment, if the task instructions include a video icon, the user may initiate the video by pointing a finger or an object, such as a pointer, at a display of the video icon as viewed through the AR glasses, and the program logic 106 runs (i.e., starts) the video in response to detecting the pointer or finger. In one embodiment, the pointer or finger is partially covered with a material that reflects a particular wavelength, such as a narrowband wavelength, and the program logic is configured to start running the video upon receiving this wavelength for a minimum time. For example, if the pointer is placed on the video icon for at least three seconds, the video is run by the program logic 106.

[0182] In further embodiments, equipment includes machines (e.g., machines that process raw materials or intermediate products, such as pumps, mixers, material conveyors (i.e., conveyor belts, pipes, etc.), ovens, automated systems or components of automated systems, robots and / or robotic systems, spectrometers, chromatographs, etc.), as well as passive plant equipment such as floors, walls, ceilings, storage units, trellises, stands, etc.

[0183] 1A , the automation system further includes a distributed control system 132 coupled to the database system 102. The distributed control system (DCS) 132 is further coupled to the sensors 112, 114, 116, the actuators 118, 120, the equipment 122, and the robot 124 and configured to control operation of and / or receive information from the sensors 112, 114, 116, the actuators 118, 120, the equipment 122, and / or the robot 124. In one embodiment, the DCS 132 includes a memory 134 configured to store one or more process parameters corresponding to the equipment 122 and / or the robot 124, as well as process control software 136, and a processor 138 configured to execute the process control software 136 to automatically control a production process in a plant. For example, the processor 138 may be configured to execute process control software for automatically controlling a production process based on measurement signals received from one or more sensors 112, 114, 116 and control signals sent to one or more actuators 118, 120.

[0184] Process parameters indicate the current state of the plant's production process, such as equipment operating parameters including equipment status (e.g., indications of faults) and values ​​of various equipment parameters such as flow rates, pressures, temperatures, etc.

[0185] The sensors are configured to measure one or more process parameters of the equipment or parameters of the plant itself, such as ambient environmental parameters, and the actuators are coupled to one or more pieces of equipment 122 and / or robots 124 to control the process parameters of the equipment and / or robots.

[0186] In another embodiment, the automation system further includes a user database 140 coupled to the DCS 132. The user database 140 is configured to store user information associated with the production process. For example, in one embodiment, the user information includes a user ID, a user role, and / or a user privilege. In one embodiment, based on the user's spatial location as well as the user information (corresponding to a particular user) received by the program logic 106 from the user database 140, the program logic 106 controls the content of the AR visualization to display in the AR glasses for a particular user of the AR glasses (i.e., which task instructions to display for a particular user, which of the contents of each task instruction and / or which task instruction content to display for a particular user). That is, the program logic 106 "filters" the received task instructions and / or the content of the task instructions based on the user of the glasses as indicated by the user information received from the user database 140.

[0187] Thus, in one embodiment of the present invention, the executable program logic 106 is configured to receive user credentials, for example, via a user interface (not shown) of the AR glasses, or alternatively, via a smartphone, other portable device, or via a scanning device that scans a personal characteristic of the potential user, such as an image of a retina or finger. The user credentials may be entered into the automated system by a user via a wireless or wired connection, for example, using a keyboard or voice command, or the user credentials may be received by the automated system via the scanning device. After receiving the user credentials, the program logic 106 authenticates the user based on matching the received user credentials with, for example, one of the user IDs.

[0188] FIG. 3 illustrates an example node 300 of a graph database, where the node represents a task instruction that may be graphically represented as an AR object as shown in FIG.

[0189] Fig. 4 shows a flow chart of a method for operating an automation system. For example, the method can be implemented based on the automation system shown in Fig. 1A. The method includes a step 402 of providing an automation system. The automation system includes an augmented reality system including a database system 102 storing data and spatial coordinates associated with each of a plurality of devices, augmented reality glasses 104 configured to display the data, and executable program logic 106 operably coupled to the augmented reality glasses, for example as described in the example with reference to Fig. 1A. The augmented reality system may include further components, for example a control system 132 and / or one or more of the devices.

[0190] Next, in step 404, the executable program logic 106 receives the most recent spatial location of the user wearing the AR glasses from the positioning system 108. The location information may be received over a network 107, as shown in Figure 1A. In other embodiments, the program logic 106 and the positioning system 108 may be integral components of the AR glasses, and the user's location information may be received over a local interface, for example a bus.

[0191] Next, in step 406, the database system determines the data of the device in proximity to the most recent spatial location of the user wearing the glasses. The determination can be performed automatically or upon request of the program logic 106, for example.

[0192] Next, in step 408, the program logic 106 receives the determined data of the device from the database system. For example, this can be performed in response to sending a request to the database system or based on a subscription of the program logic 106 to a messaging service provided by a streaming interface of the database system 102. The sent request and / or subscription includes the latest location information of the user wearing the glasses, thereby enabling the database system in step 406 to identify devices that are in sufficient spatial proximity to the user wearing the AR glasses.

[0193] Next, in step 410, the executable program logic 106 causes the AR glasses to display at least some of the received data, e.g., a GUI having instructions for operating or maintaining the equipment. The displayed data allows the user to control and / or maintain the equipment, thus also enabling an automated production process.

[0194] FIG. 5 shows an exemplary GUI displayed to a user assigned the "operator" role. For example, the operator may wear AR glasses and view the real equipment. Task instructions A1-A6 are shown by the AR glasses with associated x / y / z coordinates. For example, the AR glasses may use the xy coordinates of different equipment, such as an adding unit, a conveying and heating unit, or a packaging unit, to display respective instruction manuals or videos at a predefined distance relative to the real-world location of the real-world equipment.

[0195] For example, a user wearing the glasses may have a registered user account in a user database, and the user's user ID is associated with one or more user roles, in this case the "operator" role. Additionally or alternatively, each user role may optionally include one or more user privileges.

[0196] Thus, the executable program logic 106 is configured to select the content of the received data to be displayed (e.g., “filter” the content of the received task instructions) based on one or more user roles and / or one or more user privileges associated with the user ID. For example, the one or more user roles include only the operator role, the maintenance role, the engineering role, the administrative role, and the guest role, as a non-limiting exemplary embodiment. The one or more user privileges and / or roles limit and determine what may be displayed in the AR world through the AR glasses. For example, the user privileges for the engineering role may define which equipment in the plant a user associated with a user ID having the role of engineer can access, in the sense of viewing data corresponding to the equipment through the display of the respective task instructions. A user associated with a user ID may have two or more roles and two or more user privileges.

[0197] In the example shown in FIG. 5, a user wearing the AR glasses is assigned the role of “operator”. As a result, the executable program logic causes the AR glasses to display a view 500, where the user views an instruction manual A1 on how to wash the filter when approaching the dosing unit, and may further view additional task instructions, and respective instruction videos A2, A3. When the user approaches the conveying and heating unit, the user will view task instruction A7 through the AR glasses. Similarly, when the user approaches the packaging equipment, the user will view task instructions A4-A6. Tasks A1-A7 relate to standard tasks that must be performed in the normal operating mode of the industrial plant.

[0198] 6 illustrates an exemplary GUI 600 displayed via AR glasses to a user assigned the role of "maintenance worker." When the maintenance worker puts on the AR glasses and looks at the actual equipment, they will see task instructions related to maintenance rather than operational tasks. For example, when the user approaches the addition unit, they will see task instruction B1 for changing a defective pressure sensor 1234, and / or when the user approaches the transfer and heating unit, they will see instructions B2 on how to inspect the heater.

[0199] FIG. 7 illustrates the use of spatial anchors to show the avatars of remote users.

[0200] The remote human operator 702 may be an expert for maintaining equipment, such as, for example, a complex machine 708. Both the machine and the local operator 704 are located in an industrial plant in city B in country B. The remote user 702 is located in a different location, for example, city A in country A. The augmented reality system 101 comprises executable instructions in the form of an augmented reality application (AR application). Additionally, the AR system 101 comprises a virtual reality application 701 ("VR application"). The VR application is interoperable with the AR application. The AR system 101 enables the remote user to assist a local colleague 704 in various operations and / or maintenance tasks performed locally with or at the equipment 708. The AR system is configured to create an avatar 706 for the remote user and place the avatar at a defined location, for example, in proximity to the machine 708 that requires the user's assistance.

[0201] According to a preferred embodiment, the AR system comprises a VR application configured to generate a virtual reality for a remote user that allows the remote user to see the local user 704 and / or other real-world objects in an industrial plant, e.g., a machine 708, as perceived from the perspective of an avatar 706. For example, the remote user can see live values ​​of a production line and a machine via a robot equipped with a camera and / or further sensors, placed in the same position and orientation as his avatar 706. In some embodiments, the robot can be remotely controlled by the remote user 702, and the remote user can perform actions defined by the robot. The avatar has a defined position in a 3D environment that is generated by the AR application 106 and displayed to the local user 704 via the AR glasses 104. It also defines the local human user 704 (AR user) and the machine 708 in a position in the 3D environment that is used as a coordinate system for augmented reality ("mixed reality"). For example, the executable program logic 106 implemented as an AR application program is operatively coupled to the AR glasses 104 and also to the VR application 701 of the remote user. The AR system 101 is configured to ensure that the VR coordinate system seen by the remote user and the AR coordinate system seen by the local user 704 through the AR glasses have the same coordinate system as a basis for positioning of virtual objects (avatars, holograms, GUIs, etc.). The AR application 106 and the VR application 701 receive position information of virtual objects displayed in the augmented / virtual reality from the same database system 102. According to some embodiments, the VR application program generates a virtual reality ("manufacturing metaverse") in which the avatars of the local human operators 704 are shown with defined optical representations (e.g., only names, or even 3D avatar shapes) so that the remote users can see the local users in the VR application (not shown).Conversely, the local user (AR user) 704 can see the avatar 706 of the remote user 702 as a hologram (e.g., as a name only or as a 3D avatar shape) within the “manufacturing metaverse” that is presented as augmented reality to the local user 704 via the AR glasses 104.

[0202] According to an embodiment, the AR glasses 104 comprise a microphone 710 and an audio output interface 712, for example a speaker, for providing a user interface to the local user. Similarly, the VR application 701 may comprise a user interface including a microphone and an audio output interface for enabling the remote user 702 to interact with the VR application. The AR application program 106 operatively coupled to the AR glasses of the local user interoperates with the VR application so that the remote user 702 and the local user 704 can talk to each other. Preferably, the AR application uses position information of the local user and of the remote user's avatar in the shared coordinate system to control the volume of the audio output interface of the AR glasses, i.e., the closer the local user 704 is to the avatar 706, the louder the volume. Similarly, the sensitivity of the microphone may be adapted depending on the distance between the two users in the shared coordinate system, i.e., the greater the distance, the lower the microphone sensitivity.

[0203] Thus, the AR system according to the illustrated embodiment with AR and VR applications allows two users to collaborate using a shared coordinate system, also referred to as the "manufacturing metaverse." The manufacturing metaverse consists of a virtual coordinate system shared by all users and AR objects in the metaverse, whereby this virtual coordinate system is mapped onto real-world objects 708 of the industrial plant and presented as an overlay. This allows an experienced remote user to experience the same or a similar visual context as a guided local user.

[0204] According to a first scenario, the production of thermoplastic urethane (TPU) is taking place at location B. The production line has been modified to adapt it to a new product that has never been produced by this production line before. A remote operator is an expert in this type of production and lives at location A (country A, city A). The remote user guides the local operator as to when and which valves to check. The remote user also informs the local user of important things that the local user should pay attention to in order to ensure a proper and smooth production. Furthermore, the remote user may guide the local user as to the appropriate operating steps in the right situations and at the right times.

[0205] According to the second scenario, TPU production is already taking place at location B. During the night shift, the work coordinator for that night shift falls ill. An experienced human operator at a remote location A (country A, city A, other time zone) can support during the shift for the production at location B. That operator uses his avatar for direct collaboration with one or more local human operators. The remote user can coordinate the work of the local workers by creating processes / tasks that can be executed by the local user (AR user).

[0206] According to the third scenario, an autonomous TPU production is taking place at location B. Under normal / standard conditions, the line can operate fully autonomously. Manual operations are only necessary if an unexpected event occurs. This management job is performed remotely through the manufacturing metaverse with a VR application. One or more robots are coordinated and triggered by a remote human operator by creating / using the appropriate processes / tasks for the robots. If the remote human operator needs to see real pictures and / or video streams of the local situation, the remote user is enabled through the VR and AR applications associated with the control program of the robot to navigate to the equipment of interest. The robot is equipped with a camera and controlled to take the same position and orientation as the position of the remote user's avatar. When the robot reaches this position, the robot's camera takes an image and / or video of the equipment in front of the avatar / robot and transfers the image or video to the VR application. The remote user will see the image and / or video through the virtual reality created by the VR application. The robot is controlled to take images and / or videos from the same position and orientation as the remote user's avatar in the industrial plant's mixed reality coordinate system, so that the photos and images show the equipment of interest from the same perspective as a local human operator at said position and orientation.

[0207] Embodiments of the invention may be used in many other scenarios and industries, for example, in the automotive manufacturing industry, allowing experienced engineers to assist colleagues working at other sites in the automotive manufacturing company.

[0208] Similarly, AR systems can be used in the healthcare business, enabling medical professionals or doctors to assist other doctors working in other hospitals.

[0209] In some use case scenarios, the robot is not only used to acquire images or videos, but can also be used to solve problems under the control of a remote user. For example, the robot may be used to perform maintenance tasks in locations that are dangerous for human operators, for example in the context of deep-sea construction work, or tasks performed in environments that are contaminated or at risk of being contaminated with radioactive or toxic chemicals.

[0210] According to some embodiments, spatial anchors are used to place virtual objects at defined positions and orientations relative to real-world objects, such as the device 708. For example, it may be desirable to display a GUI hologram that allows a user to monitor and control the device 708 at a distance of about 40 cm in front of the device.

[0211] To ensure that the local user 704 always sees the GUI hologram at this defined position in the augmented reality, regardless of the user's 704 current location, the executable program logic 106 generates and displays the GUI hologram at the spatial anchor or at a defined distance relative to the spatial anchor. According to some embodiments, the spatial anchor (i.e. at least the anchor ID and the anchor coordinates) are stored in a database system to be accessible to the VR application 701. The VR application is configured to read the stored spatial anchor and generate and display a GUI hologram of the same virtual object, e.g., the equipment 708, at said spatial anchor or with the aid of a defined distance relative to the spatial anchor.

[0212] According to some embodiments, spatial anchors are defined and created by placing machine-readable codes, e.g., QR codes 711, 713, 714, at various locations in an industrial plant. The AR glasses 104 may include a camera that takes a digital image of the machine-readable code, extracts the anchor ID encoded therein, creates anchors with the coordinates of the machine-readable code in a real-world coordinate system, and stores these anchors in a database system. Alternatively, the user 704 may create a spatial anchor by performing an anchor creation gesture at a desired location in the real world. This gesture is captured by the camera of the AR glasses, and the spatial anchor is likewise created and stored by the executable program logic 106. The anchor ID may be automatically attached upon creation of the spatial anchor.

[0213] 8 illustrates a coordinate system 800 used by a VR application 701 to generate a virtual representation 808 of real-world equipment 708 of an industrial plant and display the virtual representation to a remote user 702 via virtual reality display technology. This virtual reality coordinate system 800 may further comprise a virtual representation 804, e.g., an avatar, of a local user 704 operating in spatial proximity to the real-world objects 704. The distance of the virtual representations 808, 804 of the equipment 708 and the local user 704 in the coordinate system 800, which corresponds to and reflects the actual distance of the real-world objects, may be represented via a spatial mesh that overlays the equipment of the equipment 708 and the local user 704 in the real-world industrial plant. Additionally, the remote user 702 may be represented in this coordinate system 800 as a virtual entity, e.g., an avatar 706, that can be seen by the local user 704 via AR glasses. FIG. 8 shows a coordinate system shared by the virtual reality generated by a VR application and the AR reality generated by an AR application, but preferably the remote user views objects and avatars in coordinate system 800 from the perspective (position and orientation) of the user's avatar 706 (not shown).

[0214] FIG. 9 shows a flowchart of a method for operating an automation system for controlling an automated production process of an industrial plant. For example, the method can be implemented based on the automation system shown in FIG. 1A. The automation system includes an augmented reality system including augmented reality glasses 104 configured to display data, and an executable program logic 106 operably coupled to the augmented reality glasses, for example as exemplarily described with reference to FIG. 1A. The augmented reality system may also include a database system 102 that stores data and spatial coordinates associated with each of the plurality of pieces of equipment. The augmented reality system may also include further components, for example a control system 132 and / or one or more of the equipment.

[0215] In step 902, the executable program logic receives an up-to-date spatial location of a user wearing the augmented reality glasses from the positioning system 108. The location information may be received over a network 107, as shown in FIG. 1A. In other embodiments, the program logic 106 and the positioning system 108 may be integral components of the AR glasses, and the user's location information may be received over a local interface, e.g., a bus.

[0216] In step 904, the executable program logic determines the coordinates of the boundary space proximate to the user wearing the AR glasses. The executable program logic 106 is configured to determine the coordinates of the boundary space proximate to the user based on one or more of the following boundary space parameters: the user's most recent spatial location, the user's line of sight, the number of devices proximate to or within the user's line of sight (e.g., density of devices proximate to and / or within the user's line of sight), the number and / or data type of virtual objects corresponding to devices proximate to and / or within the user's line of sight, the configuration of the walls, floors, and / or ceilings of the plant base proximate to the user, and one or more user roles assigned to the user wearing the AR glasses 104. Thus, the boundary space is defined by coordinates that define a boundary surface of the boundary space determined by the program logic. The program logic may also be configured to interpolate between the determined coordinates to add more coordinates that define a boundary surface of the boundary space with a higher resolution.

[0217] In step 906, the executable program logic displays one or more virtual objects to a wearer of the AR glasses via the AR glasses if and only if the coordinates of the virtual objects are within the boundary space. The displayed virtual objects comprise data associated with monitoring and / or controlling the automated production process. The displayed virtual objects include data such as text, video, image, audio, or alarm data. For example, in one embodiment, the program logic transmits the latest spatial location of the user wearing the AR glasses and the coordinates of the boundary space (e.g., the coordinates of the boundary surface surrounding the boundary space) to the database system 102, and receives from the database system a virtual object having coordinates that are contained within the boundary space defined by the coordinates of the boundary space. Thus, the user (e.g., wearer) of the AR glasses sees an overlay of the real-world objects of the plant and the virtual objects contained within the boundary space. The user can then use the text, image, video, audio, and / or alarm data corresponding to the displayed virtual objects to maintain and / or control the respective equipment to which the virtual objects correspond, and / or to obtain operating status, parameter settings, and / or to educate himself / herself on how the equipment operates and / or on the role of different equipment in the production process.

[0218] In step 908, the executable program logic continuously updates the three-dimensional boundary space and the coordinates of the virtual objects displayed therein according to the latest spatial position of the user wearing the AR glasses. Furthermore, the program logic may also be configured to receive data indicating the orientation of the AR glasses and / or the orientation of the eyes, and to use this additional data to determine and update the coordinates of the boundary space according to the additional data. Thus, when a user wearing or operating the AR glasses moves to a different spatial position and / or when the user is stationary with respect to the spatial position but changes the line of sight through the glasses by changing the orientation of the AR glasses (e.g., a user wearing the AR glasses as a headpiece moves the headpiece by the movement of the user's head), the program logic determines a new boundary space. The program logic 106 then displays one or more virtual objects to the wearer of the glasses via the AR glasses 104 if and only if the coordinates of the virtual objects are within the new boundary space. The virtual objects displayed in the new (i.e., updated) boundary space may include some of the virtual objects displayed in the previous boundary space depending on the overlap between the new boundary space and the previous boundary space.

[0219] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0220] Embodiments of the present disclosure may be systems, methods, products, and / or computer program products. The computer program product may include computer-readable storage medium(s) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0221] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), static random access memories (SRAMs), portable compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), memory sticks, floppy disks, mechanically encoded devices such as punch cards or ridge structures in grooves with instructions recorded on them, and any suitable combinations of the above. As used herein, a computer-readable storage medium should not be construed as being a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

Claims

1. An automation system (100) for controlling an automated production process of an industrial plant, said automation system comprising an augmented reality system (101), The augmented reality system (101) Augmented reality glasses (104); executable program logic (106) coupled to the augmented reality glasses; Including, The executable program logic (106) receiving from a positioning system (108) the latest spatial position of a user wearing said augmented reality glasses; dynamically determining the coordinates of a boundary space (150, 152) proximate to the user wearing the AR glasses; displaying one or more virtual objects (115, 118) via the AR glasses to the user wearing the AR glasses if and only if coordinates of the virtual objects are within the boundary space, wherein the displayed virtual objects comprise data associated with monitoring and / or control of the automated production process; o continuously updating the coordinates of the boundary space and of the virtual objects displayed within the boundary space according to the latest spatial position of the user wearing the AR glasses; It is configured as follows: An automated system (100).

2. 2. The automation system of claim 1, wherein the industrial plant includes a plurality of pieces of equipment (122, 124) for performing the production process, the plurality of pieces of equipment being spatially distributed within the industrial plant, the virtual objects including graphical and / or audio representations of data that enable the user to operate and / or maintain the plurality of pieces of equipment, and the coordinates of each piece of equipment of the plurality of pieces of equipment define the coordinates of at least one of the virtual objects comprising the data that enable the user to operate and / or maintain the piece of equipment.

3. 3. The automated system of claim 1 or 2, wherein the executable program logic (106) is configured to dynamically determine the coordinates of the boundary space depending on a number of devices and / or a corresponding number of virtual objects comprising data of the devices that are in spatial proximity to the user wearing the AR glasses.

4. 3. The automation system of claim 1, wherein the industrial plant comprises a plurality of different rooms and / or compartments having different heights, and the executable program logic (106) is configured to dynamically determine the coordinates of the bounded space, in particular the coordinates corresponding to the height, depending on the height of the room and / or compartment in which the latest spatial position of the user wearing the AR glasses is located.

5. the executable program logic comprising: - determining the height of the room or compartment by receiving measurement data from one or more sensors worn by the user; or determining the height of the room or compartment in response to a room ID or compartment ID detected by a sensor worn by the user; The automated system of claim 4 further configured to:

6. 6. The automated system of claim 5, wherein the executable program logic is further configured to determine the height of the room or compartment as a function of a room ID or compartment ID based on an assignment table that assigns heights to room IDs and / or compartment IDs.

7. 3. The automation system of claim 1, wherein the automation system comprises a user database, the user wearing the AR glasses being a registered user, the user database comprising an assignment of each registered user to one or more user roles, and the executable program logic (106) is configured to dynamically determine the coordinates, in particular the coordinates corresponding to a height, a size, and / or a shape of the bounded space, depending on the one or more user roles assigned to the user wearing the AR glasses.

8. 3. The automation system of claim 1, wherein the executable program logic is configured to dynamically determine the coordinates, in particular the coordinates corresponding to a height, a size, and / or a shape of the bounded space, such that the user wearing the AR glasses is presented with virtual objects associated with equipment in multiple different rooms and / or floors of the industrial plant.

9. 8. The automation system of claim 7, wherein the user roles comprise a maintenance user role and an operations user role, and wherein the executable program logic (106) is configured to dynamically determine the coordinates of the bounded space, in particular the coordinates corresponding to a height, such that the user wearing the AR glasses is presented with virtual objects associated with equipment in multiple different rooms and / or floors of the industrial plant when the user has a maintenance user role, but not when the user has an operator role.

10. 3. The automation system of claim 1 or 2, wherein at least some of the virtual objects are graphical representations of respective ones of the equipment represented by the virtual objects, the graphical representations comprising image-based and / or video-based views of the ones of the equipment.

11. 3. The automated system of claim 1, wherein the boundary space is one of two or more data type-specific boundary spaces, the virtual object comprises an object of a data type selected from the group consisting of video, image, text, audio file, and alarm message, and the executable program logic (106) is configured to dynamically determine the coordinates of the data type-specific boundary space depending on a data type of the virtual object displayed to the user wearing the AR glasses.

12. 12. The automation system of claim 11, wherein the executable program logic is configured to determine an alarm boundary space for the alarm message and at least a further boundary space for one of the other data types, the alarm boundary space being larger than the further boundary space.

13. The boundary space surrounds the user wearing the AR glasses, or The boundary space is within the field of view of the user wearing the AR glasses and does not surround the user wearing the AR glasses; 3. The automated system according to claim 1 or 2.

14. the executable program logic comprising: receiving data indicative of the location of one or more physical objects, each physical object being, among other things, equipment used in said automated production process or material used, modified, or produced by said production process; - determining a line of sight between the user wearing the glasses and one of the virtual objects; Displaying the one virtual object through the AR glasses if and only if the virtual object is within the boundary space and none of the physical objects is located within a line of sight between the user wearing the glasses and the virtual object. The automation system according to claim 1 or 2, configured to:

15. 3. The automated system of claim 1, wherein the executable program logic is configured to receive data indicating an orientation of the AR glasses and to dynamically determine and continuously update the coordinates of the boundary space in response to the data indicating the orientation of the AR glasses.

16. 3. The automated system of claim 1, wherein the executable program logic is configured to receive data indicating an eye orientation of the user wearing the AR glasses, and to dynamically determine and continuously update the coordinates of the boundary space in response to an eye orientation of the user wearing the AR glasses.

17. 1. A method for controlling an automation system (100) for controlling an automated production process of an industrial plant, the automation system comprising an augmented reality system (101) including augmented reality glasses (104) and executable program logic (106) coupled to the augmented reality glasses, the method comprising: - receiving, by said executable program logic, from a positioning system (108) the most recent spatial position of a user wearing said augmented reality glasses; - dynamically determining, by the executable program logic, coordinates of a boundary space proximate to the user wearing the AR glasses; - displaying, by the executable program logic, one or more virtual objects via the AR glasses to the wearer of the AR glasses if, and only if, coordinates of the virtual objects are within the boundary space, wherein the displayed virtual objects comprise data associated with monitoring and / or control of the automated production process; - continuously updating, by the executable program logic, the coordinates of the boundary space and of the virtual object displayed within the boundary space according to the most recent spatial position of the user wearing the AR glasses; A method comprising:

18. The executable program logic (106) configured to receive data indicative of the orientation of the AR glasses and to dynamically determine and continuously update the coordinates of the boundary space in response to the data indicative of the orientation of the AR glasses; and / or configured to receive data indicative of an eye orientation of the user wearing the AR glasses, and to dynamically determine and continuously update the coordinates of the boundary space depending on the eye orientation of the user wearing the AR glasses; The method of claim 17, wherein the method is configured to:

Images