Augmented reality-based automation system with task manager

Abstract

An automation system for controlling an automated production process of an industrial plant is provided. The industrial plant includes a plurality of equipment for performing a production process. The automation system includes an augmented reality system including augmented reality glasses, executable logic, and a database system including a task manager configured to store a list of processes associated with respective equipment, wherein each process includes one or more tasks to be performed with respect to the equipment. The executable program logic is configured to receive, from the database system, the list of processes for display to a user of the AR glasses, receive a selection of a displayed process assigned to the user, send the selected process to the database system, and receive, from the database system, additional data corresponding to the selected process.

Description

[0001] manual Technical Field

[0002] Certain embodiments of the present invention relate to the field of automation systems. More specifically, certain embodiments of the present invention relate to augmented reality-based automation systems applied to industrial plants. Background Art

[0003] In an industrial plant, input materials are processed to manufacture one or more products. The properties of the manufactured products therefore depend on the manufacturing parameters and the input materials.

[0004] In the process industry, in particular in an industrial plant such as a chemical production plant or a biological production plant, a production process is used to process one or more input materials to produce one or more chemical products or biological products. The production environment in the process industry can be complex, so the properties of the product may change according to changes in the production parameters that affect the properties. Often, the dependence of the properties on the production parameters may be complex and intertwined with further dependence on one or more combinations of specific parameters. Therefore, it can be challenging to produce chemical products or biological products with consistent and / or predictable quality.

[0005] In particular, product quality and / or production stability depends on the operator having up-to-date information about the state of the plant, including quick and easy access to the operating characteristics of the plant equipment and supporting infrastructure, as well as the values ​​of current process parameters corresponding to the respective equipment, ambient plant parameters, and / or parameters associated with the feedstock and intermediate products produced during the processing of the materials. Information about the state of the plant may also include the maintenance history of the equipment and information about how each piece of equipment is operating, including the range of operating parameters for safely operating the equipment.

[0006] The operation and maintenance of production facilities often requires specialized knowledge of machines and manufacturing processes. However, there is often a lack of employees with the appropriate knowledge. At the very least, it may take some time for properly trained maintenance personnel to arrive on site, for example when a machine has a technical problem.

[0007] There is therefore a need for improved methods of controlling and / or maintaining automated factory production processes to provide products of consistent and / or predictable quality. Summary of the invention

[0008] At least some of the problems inherent in the prior art will be solved by the subject matter of the attached independent claims. At least some of the further advantageous alternatives will be outlined in the dependent claims.

[0009] Various embodiments provide a system for controlling an automated production process of an industrial plant as described in the subject matter of the independent claim. Advantageous embodiments are described in the dependent claims. If the embodiments of the invention are not mutually exclusive, they can be freely combined with each other.

[0010] In one aspect, the present invention relates to an automation system for controlling an automated production process of an industrial plant. The industrial plant includes a plurality of equipment for performing the production process, wherein the plurality of equipment is spatially distributed in the industrial plant. The automation system includes an augmented reality system, which includes: a database system configured to store data associated with corresponding equipment in the plurality of equipment, wherein the data includes a process list, and wherein each process includes one or more tasks to be performed relative to one or more equipment; augmented reality glasses, the augmented reality glasses are configured to display the data; and executable program logic, the executable program logic is coupled to the augmented reality glasses and the database system. The database system includes a task manager configured to manage processes relative to one or more equipment. The process list is managed by the task manager. The executable program logic is configured to receive the process list from the database system for display to a user of the AR glasses, wherein at least one of the processes is assigned to the user of the AR glasses. The program logic is further configured to: receive a selection made by the user of a displayed process assigned to the user, send the selected process to the database system, and receive additional data corresponding to the selected process from the database system for display to the user of the AR glasses. Each process includes a user-selectable task link. The task link is configured to display a task in the one or more tasks associated with the process in the AR glasses when selected by the user. The system may also be configured to receive a hands-free selection of the task link by the user of the AR glasses. In one embodiment, the additional data includes the one or more tasks.

[0011] These features have the advantage of displaying a limited amount of information to the user (i.e., the wearer) of the AR glasses, so that the user is not overwhelmed by too much information, and in particular guides only those processes assigned to the user to be displayed to the user. The additional information can then be displayed to the user only by the user performing a hands-free operation for selecting a virtual object linked to the corresponding secondary virtual object (i.e., task) (i.e., user-selectable task link). That is, when the user performs a hands-free selection of a virtual object linked to the corresponding secondary virtual object, the corresponding secondary virtual object containing the additional information is displayed.

[0012] In addition, selection of the virtual object linked to the secondary virtual object can advantageously enable the user to trigger an action associated with the display of the secondary virtual object (i.e., the displayed task) in a hands-free manner, such as routing the user to the location of the equipment in the factory for the user to perform the task on the equipment, or routing a robotic unit to the location of the equipment in the factory for the robotic unit to perform the task, or providing a navigation route that avoids collisions between the user of the AR glasses and the robotic unit or delays caused by movement of the robotic unit in the factory, or providing a navigation route that avoids hazardous or dangerous areas in the factory.

[0013] In addition, the hands-free option advantageously allows the user to obtain additional information in the form of video data, audio data, text data, image data and / or alarm data while using his / her hands to perform other tasks, such as adjusting machine parameters, throwing switches, repairing equipment and / or staying safe in hazardous areas by holding onto railings with one hand, etc. (e.g., for balanced maintenance). Furthermore, the user does not need to carry instruction manuals and maintenance records when inspecting and controlling production lines and production processes.

[0014] The hands-free option also advantageously allows the user to record the tasks performed through the AR glasses in the form of capturing photos, videos, or uploading documents.

[0015] In one aspect, the present invention relates to an automation system for controlling an automated production process of an industrial plant. The industrial plant includes a plurality of equipment for performing the production process, wherein the plurality of equipment is spatially distributed in the industrial plant, and wherein each of the plurality of equipment is associated with one or more corresponding virtual objects. The automation system includes an augmented reality system, which includes augmented reality glasses and executable program logic coupled to the augmented reality glasses. The executable program logic is configured to: receive the spatial position of a user wearing the augmented reality glasses from a positioning system; display one or more virtual objects associated with one or more equipment near the spatial position of the user via the augmented reality glasses, wherein at least one of the one or more virtual objects is linked to at least one corresponding secondary virtual object, and wherein the at least one corresponding secondary virtual object is stored in a database system; receive a signal from the augmented reality glasses indicating the user's hands-free selection of a virtual object linked to a corresponding secondary virtual object; and display the corresponding secondary virtual object via the augmented reality glasses and / or trigger an action associated with the selected virtual object.

[0016] These features have the advantage of displaying a limited amount of information to the user (i.e., the wearer) of the AR glasses, so that the user is not overwhelmed by too much information. Additional information can be displayed to the user by the user performing a hands-free operation for selecting a virtual object linked to the corresponding secondary virtual object. That is, when the user performs a hands-free selection of a virtual object linked to the corresponding secondary virtual object, the corresponding secondary virtual object containing the additional information is displayed. In addition, the selection of a virtual object linked to a secondary virtual object can advantageously enable the user to trigger an action associated with the virtual object in a hands-free manner, such as starting or stopping the equipment associated with the selected virtual object or modifying the process steps or operating parameters of the equipment associated with the selected virtual object. In addition, the hands-free selection advantageously allows the user to obtain additional information in the form of video data, audio data, text data, image data, and / or alarm data while using his / her hands to perform other tasks, such as adjusting machine parameters, throwing switches, repairing equipment, and / or staying safe in dangerous areas by grabbing a railing with one hand (e.g., for balance maintenance). In addition, the user does not need to carry instructions for use and maintenance records when inspecting and controlling production lines and production processes.

[0017] According to an embodiment, the executable program logic is configured to: determine the number of at least one virtual object linked to the at least one corresponding secondary virtual object and associated with one or more equipment near the spatial location of the user; dynamically assign a different color from a predefined color set of colors to each virtual object linked to the corresponding secondary virtual object and display each virtual object linked to the corresponding secondary virtual object with the dynamically assigned different color; and if the number of virtual objects linked to the corresponding secondary virtual object is greater than the number of colors in the predefined color set, only display those virtual objects linked to the corresponding secondary virtual object that are assigned different colors.

[0018] These features have the advantage of further limiting the amount of information presented to the user (i.e., the wearer) of the AR glasses so that the user does not feel overwhelmed, by further limiting the display to the user to only those virtual objects that are linked to secondary virtual objects, which can be assigned colors that differ from a limited number of predefined colors.

[0019] According to other embodiments, the hands-free selection includes a voice selection of the virtual object. For example, the augmented reality glasses and / or the positioning system may include a voice detection unit configured to detect the voice selection and convert the voice selection into the signal indicating the user's hands-free selection of the virtual object.

[0020] These features have the advantage of using an uncomplicated hands-free selection process that enables the user to say a single simple word, such as the color of the virtual object the user wishes to select. The system can be advantageously used in multiple countries by users who speak different languages, because most users either know or can easily learn the English words for easily recognizable standard colors, such as primary colors, or at least some of the primary and secondary colors, and black and white. In addition, the system does not require training of voice recognition software, because the voice commands are simple, monosyllabic or disyllabic, and in other embodiments, two words or a word and a number.

[0021] In one aspect, an automation system for controlling an automated production process of an industrial plant is provided. The automation system includes an augmented reality system, and the augmented reality system includes augmented reality glasses and executable program logic coupled to the augmented reality glasses. The executable program logic is configured to: receive the latest spatial position of a user wearing the augmented reality glasses from a positioning system; dynamically determine the coordinates of a boundary space close to the user wearing the AR glasses; display one or more virtual objects to the wearer of the AR glasses via the AR glasses when and only when the coordinates of the virtual object are within the boundary space, wherein the displayed virtual object includes data associated with monitoring and / or controlling the automated production process; and continuously update the boundary space and the coordinates of the virtual object displayed therein according to the latest spatial position of the user wearing the AR glasses.

[0022] According to an embodiment, the executable program logic is configured to receive data indicating the orientation of the AR glasses, and is configured to dynamically determine and continuously update the coordinates of the bounding space based on the data indicating the orientation of the AR glasses.

[0023] According to an embodiment, the executable program logic is configured to receive data indicating the eye orientation of the user wearing the AR glasses, and is configured to dynamically determine and continuously update the coordinates of the boundary space based on the eye orientation of the user wearing the AR glasses.

[0024] These features have the advantage of quickly and efficiently enabling the user (i.e., the wearer) of the AR glasses to obtain the latest virtual objects contained within a bounding space that changes dynamically according to the user's variable spatial position and / or the variable orientation of the AR glasses, as well as the advantage of defining (i.e., "filtering") those objects for display by using the bounding space defined by the determined coordinates. In addition, the virtual objects represented by data (such as text, images, videos, alarms and / or audio data) overlay (i.e., enhance) the view of the factory through the AR glasses, thereby enabling the user to effectively control and / or maintain the automation process in a hands-free manner (i.e., the user does not need to carry operating instructions and maintenance records when inspecting and controlling production lines and production processes). In addition, the amount of data will not overwhelm the user because, according to an embodiment, the executable program logic is configured to limit the data (i.e., virtual objects) displayed to the user via the augmented reality glasses (AR glasses) to only those virtual objects contained within the bounding space.

[0025] In an embodiment, the executable program logic is configured to dynamically determine the coordinates of the boundary space based on the number of equipment and / or the corresponding number of virtual objects including data of the equipment that are spatially close to the user wearing the AR glasses, and / or based on the dimensions of the factory infrastructure close to the user (such as the height of the room and / or compartment where the most recent spatial position of the user wearing the AR glasses is located), and / or based on one or more user roles assigned to the user wearing the AR glasses, and / or based on whether the boundary space is defined as a data type specific boundary space, wherein the boundary space is defined to only include virtual objects including objects of one or more data types selected from the group consisting of video, image, text, audio file, and alarm message.

[0026] According to some embodiments, the size and / or shape of the boundary space is determined according to the data type of the virtual object to be displayed. For each data type, a corresponding data type specific boundary space can be generated and used to determine whether a specific virtual object should be output via the AR glasses.

[0027] A compartment may be, for example, a cavity associated with equipment, such as a tank or a machine. A user may need to enter a compartment for maintenance purposes. In this case, it may be helpful for the bounding space to reflect the interior dimensions of the compartment, so as to avoid overburdening the user with virtual objects including information about nearby machines or machine parts that the user cannot operate anyway from his current location.

[0028] For example, a cubicle may be marked with a QR code or an RFID tag or similar tag that includes a room ID assigned to the cubicle. The AR glasses or another electronic device of the user (e.g., the user's smartphone) may include a reading device (camera, RFID reader, etc.) that is configured to read the room ID from the tag or marker when the user wearing the AR glasses enters the cubicle. The room ID is forwarded to the executable program logic to enable the program logic to determine the size of the cubicle entered by the user and to create and use a bounding space whose coordinates are determined based on the size or ID of the cubicle.

[0029] According to an embodiment, the executable program logic is configured to dynamically determine the coordinates of the bounding space according to the number of equipment and / or the corresponding number of virtual objects including data of the equipment that is spatially close to the user wearing the AR glasses. The dynamic determination of the coordinates of the bounding space particularly includes determining the size and / or shape according to the number of equipment or the corresponding number of virtual objects.

[0030] These features, individually or in various combinations with one another, have the following advantages: providing a specifically customized bounding space to provide a more refined presentation of virtual objects to the user, such as providing a bounding space containing only the most urgent virtual objects for display to the user given the user's current latest spatial position and / or orientation of the AR glasses, or providing a bounding space containing only those virtual objects that are most relevant to the role assigned to a particular user, including optionally displaying relevant virtual objects that are not necessarily close to the user, such as virtual objects corresponding to equipment located on different floors and / or rooms or areas of a factory, which would not be available for display to the user without customizing the bounding space. In addition, providing a bounding space that is adjustable in size and shape prevents the user from being overwhelmed by too many virtual objects, particularly relevant to providing a bounding space for users of AR glasses whose spatial location is in a high equipment density area of ​​a factory.

[0031] On the other hand, a method for controlling an automated system for controlling an automated production process of an industrial plant is provided. The automated system includes an augmented reality system, which includes augmented reality glasses and an executable program logic coupled to the augmented reality glasses. The method includes: receiving the latest spatial position of a user wearing the augmented reality glasses from a positioning system by the executable program logic; dynamically determining the coordinates of a boundary space close to the user wearing the AR glasses by the executable program logic; displaying one or more virtual objects to the wearer of the AR glasses via the AR glasses by the executable program logic when and only when the coordinates of the virtual objects are within the boundary space, wherein the displayed virtual objects include data associated with monitoring and / or controlling the automated production process; and continuously updating the three-dimensional boundary space and the coordinates of the virtual objects displayed therein according to the latest spatial position of the user wearing the AR glasses and / or according to the orientation of the AR glasses and / or according to the eye orientation ("gaze") of the user by the executable program logic.

[0032] In one aspect, an automation system for controlling an automated production process of an industrial plant is provided, the industrial plant including a plurality of equipment for executing the production process, the plurality of equipment being spatially distributed in the industrial plant. The automation system includes an augmented reality system, and the augmented reality system includes a database system configured to store data and spatial coordinates associated with corresponding equipment in the plurality of equipment, augmented reality glasses configured to display the data, and executable program logic coupled to the augmented reality glasses. The executable program logic is configured to receive the latest spatial position of a user wearing the augmented reality glasses from a positioning system, and to receive data of equipment located near the latest spatial position of the user from the database system. The executable program logic is further configured to control the augmented reality glasses to display at least some of the received data.

[0033] These features have the following advantages: data is quickly, efficiently, and securely received by the user of the AR glasses corresponding to the equipment near the user of the AR glasses, wherein the data overlays (i.e., enhances) the view of the factory through the AR glasses, thereby enabling the user to effectively control the automation process in a hands-free manner (i.e., the user does not need to carry operating instructions and maintenance records when inspecting and controlling production lines and production processes). In addition, the amount of data does not overwhelm the user because, according to an embodiment, the executable program logic is configured to limit the data displayed to the user via the augmented reality glasses (AR glasses) to data assigned to the equipment near the user's location. The executable program logic is further configured to control the augmented reality glasses to display at least some of the received data (if not all).

[0034] In one embodiment, the augmented reality system (AR system) also includes a positioning system. The positioning system is configured to sense the spatial position of the user in the factory. These features have the advantage of adapting the positioning system to the requirements of industrial factories. For example, if the factory is not an open floor plan, but partitioned, the commercially available GPS system may not be effective in such a closed indoor environment. The positioning system as a component of the augmented reality system can alleviate such problems.

[0035] For example, and according to further embodiments, the positioning system may include a spatial mapping grid covering at least a portion of the plant, an indoor positioning system, a cellular positioning system, or even a combination of one of the foregoing systems and a GPS system.

[0036] According to an embodiment, the spatial mapping grid includes spatial anchors. The spatial anchors are connected to each other. Each spatial anchor associates a coordinate system associated with the augmented reality glasses used to display data to a spatial coordinate system associated with the spatial coordinates of one or more of the plurality of equipment stored in the database system. The program logic is configured to display data of equipment located near the user's latest spatial position at a predefined relative position to the spatial anchor via the augmented reality glasses.

[0037] According to an embodiment, the augmented reality system is configured to create an avatar for at least one remote user. A remote user is a user far away from an industrial plant. The program logic is configured to display the avatar near the latest spatial position of the user wearing the augmented reality glasses via the augmented reality glasses. According to a preferred embodiment, the augmented reality system is configured to provide a two-way visual and / or acoustic communication channel between the user wearing the augmented reality glasses and the remote user. Thus, the AR glasses give the wearer the impression that he is talking directly to the avatar rather than to the remote user. For example, the program logic displays the avatar via the AR glasses and / or controls the output of the remote user's voice via the speakers of the AR glasses, so that the position of the avatar is perceived as the source of the remote user's voice.

[0038] According to some embodiments, the automation system includes a robot with a camera and / or a microphone. The robot is located in an industrial plant, so it is "local" to a user wearing the AR glasses and working in the industrial plant. The augmented reality system includes a robot control module, which is configured to control the movement and / or orientation of the robot so that the position and / or orientation of the robot reflects the position and / or orientation of the avatar in the augmented reality displayed via the AR glasses. The robot control module is configured to automatically update the position and / or orientation of the avatar and the robot according to changes in the position and / or orientation of the remote user or in response to control commands submitted by the remote user.

[0039] This may allow for seamless integration and merging of the "realities" of remote users, local users, and various physical objects (materials, equipment) into a single augmented reality displayed via AR glasses.

[0040] According to an embodiment, the augmented reality glasses include an acoustic output interface, wherein the augmented reality system is configured to provide a two-way acoustic communication channel between a user wearing the augmented reality glasses and an avatar. The program logic is configured to control the acoustic output interface so that the volume of the speech output to the avatar of the user wearing the augmented reality glasses via the acoustic output interface is negatively correlated with the distance between the user wearing the glasses and the avatar in a coordinate system associated with the augmented reality glasses used to display data.

[0041] The acoustic output interface may be, for example, a speaker in the environment of the user wearing the AR glasses, headphones or in-ear speakers that may be integrated into the AR glasses.

[0042] In one embodiment, the database system is a graph database system and / or a spatial database system and / or a streaming database system.

[0043] In another embodiment, the automation system also includes a control system. The control system can be a distributed control system. The control system includes a memory and control software storing one or more process parameters. At least some of the parameters are stored in association with one or more equipment in the equipment, such as by storing the parameters in association with the equipment ID. The control system includes a processor configured to execute process control software for automatically controlling the production process. For example, the control software can control the use of the equipment so that the production process is executed according to one or more parameters assigned to the equipment. For example, this can mean that one or more parameters in the parameter represent the desired machine state and / or product features, and the execution control makes the machine run in the desired state and / or makes the product have the desired features. For example, if the product should have a desired shape, size and / or temperature, the production process can be controlled so that any difference between the measured, observed product features and the desired product features is minimized.

[0044] According to an embodiment, the AR system is configured to generate and display a GUI via the AR glasses, wherein the GUI enables a user wearing the AR glasses to set one or more of the parameters, for example via gesture detection and gesture interpretation.

[0045] For example, the AR glasses may be equipped with a gyroscope configured to detect the orientation of the AR glasses and the user carrying the AR glasses. In addition, the AR glasses may be equipped with one or more cameras configured to acquire one or more images near the front of the user. The executable program logic may be configured to analyze the digital images acquired by the camera of the AR glasses for identifying the user's gestures. The program logic is configured to identify user actions and input, modify or delete parameters via the GUI based on the type of gesture, the location and / or direction of the gesture.

[0046] These features have the following advantages: enabling the user of the AR glasses to modify the control software of the distributed control system based on the data communicated to the user through the AR glasses to change the operation of the equipment used to provide more consistent and predictable products. For example, the user may determine that a particular sensor of the equipment or equipment part is too sensitive and may therefore incorrectly detect production errors (false positive problems). In this case, the user can set the sensitivity control parameter of the sensor to a lower value, thereby reducing the number of false positive alarms that may slow down the production process. According to another example, the user may determine that the actuator of the equipment is too weak, resulting in an unacceptable number of erroneous products. In this case, the user can use the GUI presented via the AR glasses to increase the parameters of the power / motion amplitude of the actuator that controls the equipment.

[0047] According to an embodiment, the AR system is configured to generate a GUI that selectively allows a user to set, modify, or delete parameters assigned to equipment that is within a predefined maximum distance from the user. For example, the maximum distance may be 5m, or 2m, or even 1m. This may have the advantage that only those parameters that are relevant are presented to the user when processing a particular piece of equipment or piece of equipment.

[0048] According to an embodiment, the control system is a distributed control system comprising a plurality of control software programs or modules interconnected to each other via a network.

[0049] According to another embodiment, the at least one equipment includes one or more sensors coupled to the at least one equipment and one or more actuators coupled to the at least one equipment. The one or more sensors are configured to measure one or more process parameters of the at least one equipment, wherein the one or more process parameters indicate the current state of the production process of the factory and / or indicate the current operating state or mode of the equipment. The one or more actuators are configured to control the one or more process parameters. In addition, the distributed control system is coupled to the one or more sensors and the one or more actuators, and the processor is configured to execute the process control software for automatically controlling the production process based on the measurement signal received from the one or more sensors and the control signal sent to the one or more actuators. In addition or alternatively, the one or more actuators are configured to operate according to one or more control parameters (e.g., pressure, temperature light intensity, concentration of one or more chemicals, machine operation mode, etc.).

[0050] According to another embodiment, the automation system of claim 1 includes a user database coupled to the distributed control system. The user database is configured to store user information associated with the production process. In some embodiments, the user information includes a user ID, a user role, and / or a user authority.

[0051] In yet another embodiment, the executable program logic is configured to receive user credentials from the user and authenticate the user. For example, authentication can be performed by matching the user credentials with the user ID in the user ID. The user ID of the user wearing the augmented reality glasses is associated with one or more user roles in the user roles, and each of the one or more user roles includes one or more user rights in the user rights. The executable program logic is further configured to select and display the content of at least some of the received data for display based on the one or more user roles and / or the one or more user rights associated with the user ID.

[0052] These features have the advantage of providing data only to authorized users (i.e., providing data through a secure system), and if the user is authorized, the selection of data to be viewed by the user (which may include the type of data and / or the content of the data) is further based on the one or more user roles and / or the one or more user permissions associated with the user ID of the user.

[0053] According to an embodiment, the executable program logic is configured to generate a GUI to be displayed to the user via the AR glasses, thereby dynamically determining the content of the GUI to be displayed according to the role assigned to the user wearing the AR glasses. The executable program logic is configured to generate a first type of GUI for the user if the user is within a predefined maximum distance from a specific equipment and has a first role, and is configured to generate a second type of GUI for the user if the user is within a predefined maximum distance from the specific equipment and has a second role. The first GUI and the second GUI are different, for example, including different GUI elements, displaying different information, and implementing different types of actions for maintaining and / or controlling the equipment. For example, the first user role can be an "operator", that is, a user who is obliged to control and / or monitor one or more steps in a production workflow. The second user role can be a "maintenance worker", that is, a user who is obliged to perform maintenance words to ensure that at least one equipment is working properly. For example, a maintenance worker may need to repair a faulty machine, replace worn parts of equipment, perform some quality tests on equipment, etc. Generally speaking, a maintenance worker is responsible for technically complex tasks that may require in-depth technical knowledge of the equipment, which is usually not available to users with the "operator" role. Thus, if an “operator” user approaches a particular equipment / machine, that user will see a different GUI than the one a “maintenance worker” sees via his AR glasses when approaching the equipment / machine.

[0054] This can make the various tasks to be performed with the help of or on the equipment more efficient and error-resistant, since each user will see via the AR glasses only the information that is really needed to complete their actual task. The risk of error scenarios where a "normal operator" causes errors in the production process by accidentally modifying machine parameters that should only be changed by skilled maintenance workers can be greatly reduced, since the "operator" user will not "see" the GUI that enables it to change said machine parameters.

[0055] According to an embodiment, the executable program logic is configured to perform a query on a database based on the latest spatial location to retrieve data of equipment in the vicinity of the latest spatial location of the user from the database.

[0056] According to an embodiment, the database system is a spatial database system, wherein the spatial database system includes a spatial database. The executable program logic is configured to perform a query on the spatial database based on the latest spatial position to retrieve data of equipment near the latest spatial position of the user from the spatial database.

[0057] These features have the advantage of storing, searching, and retrieving data (including the latest spatial location of the user of the AR glasses) from a database system based on querying a spatial database. That is, data associated with a plant object close to the spatial location of the user is linked to the spatial location of the plant object, which is also stored in the database system, thereby enabling access to the corresponding data in a fast and efficient manner when querying the database. These features also have the advantage of retrieving data only upon query request, thereby saving network bandwidth.

[0058] In another embodiment, the database system is accessible via a subscription-based streaming service. The database system may be referred to as a streaming database system. It may be implemented as a spatial and / or graph-based database system.

[0059] The executable program logic is configured to receive data of equipment near the user's latest spatial location from the database via the streaming service. According to an embodiment, the executable program logic is configured to selectively receive data of equipment near the user's latest spatial location (rather than data of equipment not near the user) from the spatial database via the streaming service. For example, the database system can use a predefined or user-defined maximum distance threshold to determine whether the location of the equipment is close to the user. These features have the advantage of continuously and quickly accessing data through the streaming service, thereby eliminating the need to send and / or receive queries.

[0060] For example, a user can subscribe to a streaming service that notifies the user when at least one device is close to the user and is now within a predefined maximum distance from the user. When the user subscribes to the service, the user's current location is automatically determined and provided as a parameter of the subscription request. The database system is configured to automatically and repeatedly check whether the distance between the user who has subscribed to the notification service and the at least one device is below a maximum distance threshold. In response to the determination, the GUI, one or more GUI elements and / or parameters assigned to the at least one device are displayed to the user via the AR glasses. For example, the user's subscription to the service is automatically executed by the program logic, for example, when the user is successfully authenticated at the AR system and / or when the user approaches the production system at a distance closer than the user-defined or predefined maximum distance (e.g., less than 10m, or less than 5m, or less than 2m, or less than 1m).

[0061] According to an embodiment, the database system is configured to selectively provide data of equipment proximate to the latest spatial location of the user wearing the augmented reality glasses in response to a query (e.g., a conventional SQL query) to the executable program logic. Movement of the user relative to the equipment triggers the executable program logic to submit a new query to the database system including the new latest spatial location of the user.

[0062] According to other embodiments, the database system is configured to selectively provide data of equipment located near the latest spatial position of the user wearing the augmented reality glasses via a subscription-based streaming service. The user's movement relative to the equipment triggers the executable program logic to update the subscription with the user's new latest spatial position.

[0063] According to an embodiment, the program logic is configured to automatically update the user's subscription to the streaming service in response to a change in the user's location. For example, the AR glasses may include a GPS sensor or another component for determining the user's current location. In response to determining that the user has changed his location, the user's subscription to the streaming service of the database system is automatically updated, thereby ensuring that the user will always be able to see the data and GUI related to one or more equipment next to the user. This can have the following advantages: the AR system and its components do not have to repeatedly submit conventional SQL queries with the user's current location information. It is sufficient to update the user's subscription to one or more streaming services associated with the database system to ensure that the data displayed via the AR glasses takes into account the user's current location, thereby significantly reducing network traffic and the overall computational load of the AR system, in particular the computational load of the database system.

[0064] According to another embodiment, the spatial database system is a graph database system, which includes a graph database having multiple interconnected nodes. Each node represents one or more objects of an industrial plant, and includes spatial information associated with the one or more objects. For example, in the case where a node represents multiple objects, these objects are usually close to each other during the manufacturing process. According to the embodiment, the spatial information associated with the one or more objects may be the position information of a single object in the multiple objects, or may be the position information of each physical object in the multiple physical objects. The object is a sensor, an actuator, a raw material, or equipment (e.g., a tool, or a machine or a robot) in a plurality of equipment. The architecture of the graph database enables faster search and access to data associated with a given object in the factory using only the spatial coordinates of the user of the AR glasses in the factory. In addition, by representing the objects of the industrial plant that are close to each other and remain close to each other during the manufacturing process by a single node in the graph, the positions of all objects of the node can be monitored at least approximately, so that the position information of at least one object represented by the node is available.

[0065] According to some embodiments, the executable program logic is configured to:

[0066] - receiving coordinates of a visual object graphically representing data of the equipment to be displayed; for example, the program logic may receive coordinates, such as 3D coordinates of the position and size of a visual representation of a manual or operating instructions for said equipment, or coordinates of a virtual "TV screen" within which instructions for operating the equipment are to be displayed via the AR glasses;

[0067] - determining the user's line of sight in a real-world coordinate system; for example, the executable program logic may receive an indication of the current position and orientation of the user wearing the AR glasses from a positioning system;

[0068] - receiving from the database system the coordinates of a real-world object that is proximate to the most recent spatial position of the user wearing the augmented reality glasses; the real-world object may be a portable and / or mobile equipment, a raw material or a product. The real-world object may be an object whose current position is constantly monitored and tracked by the augmented reality system, for example, with the aid of a GPS sensor, an indoor positioning system, one or more capacitive or inductive proximity sensors, etc.; therefore, the coordinates of the real-world object indicate the real position, orientation and size of the real-world object in the real world; for example, the program logic may receive the coordinates of a virtual grid generated as an overlay of equipment on a real-world production line; the received coordinates may also be coordinates dynamically determined with the aid of a position determination sensor for equipment or other types of real-world objects;

[0069] - determining whether one or more of the virtual objects representing the data of the equipment are located along the user's line of sight and are farther away from the user than at least one of the real-world objects also located along the user's line of sight; and

[0070] - not selectively displaying the one or more virtual objects determined to be in the user's line of sight and positioned farther than at least one of the real-world objects.

[0071] This may have the advantage that if it is determined that a virtual object (eg a manual) would be obscured by a "real world object" if displayed, the virtual object is not displayed.

[0072] According to an embodiment, the executable program logic is configured to:

[0073] - receiving data indicating the position of one or more physical objects, each physical object being in particular equipment for the automated production process or material used, modified or produced by the production process;

[0074] - determining a line of sight between the user wearing the glasses and one of the virtual objects;

[0075] - Displaying the one virtual object via the AR glasses if and only if the virtual object is within the bounding space and no physical object is located in the line of sight between the user wearing the glasses and the virtual object.

[0076] This feature can prevent the user from being distracted, because it is assumed that a real-world object that is even closer to the user in the user's line of sight (visual axis) than, for example, a hologram with an educational film will require even greater attention. For example, the user may be performing repair work on the conveyor belt while the conveyor belt continues to run. As soon as the workpiece transported on the belt enters the user's viewing direction and moves in front of the hologram, the hologram is completely or partially hidden (because the AR system stops displaying the hologram or at least the part of the hologram that is hidden by the real-world object). This has the following advantages: the user can always pay attention to the moving object, so that accidents can be avoided. In addition, reading errors are also avoided, because when the AR glasses are used to overlay the hologram on a dynamically changing real-world scenery, the brightness and contrast of the background change, so the user may make mistakes when reading the data of the virtual object. It also makes the visual impression even more "immersive" for the user, because real physical objects that can be static or moving can cover and hide virtual objects, making the dynamically "hidden" virtual objects look even more real and blend more seamlessly with physical world objects.

[0077] According to an embodiment, one or more of the nodes include or are stored in association with a corresponding container.

[0078] As used herein, a "robot" is a machine that can automatically move and perform certain tasks. According to some embodiments, the robot includes a robot control module that is configured to control movement, position, orientation, and / or tasks performed by the robot. Additionally or alternatively, the robot can be operably coupled to a robot control module outside the robot. For example, the control system for an embodiment of the present invention described herein may include a robot control module or parts thereof. The robot control module is configured to receive state information from one or more sensor robots and send control commands to the robot or one or more parts of the robot, whereby the control commands determine movement, position, orientation, and / or tasks performed by the robot. One or more sensors of the robot may include a position determination sensor, such as a GPS sensor, a gyroscope and accelerometer for determining the position and orientation of the robot, a camera and a microphone for capturing images or videos and / or acoustic signals.

[0079] The coordinate system "associated with the AR glasses" is a coordinate system of augmented reality that includes real-world objects (such as machines and materials) and one or more virtual objects. In the coordinate system of augmented reality, the coordinates of real-world objects and virtual objects are mapped to the augmented reality coordinate system and displayed to the user wearing the glasses via the AR glasses, so that the virtual objects and real-world objects form part of a shared space (called augmented reality). Therefore, the coordinate system "associated with the AR glasses" can be considered to be a coordinate system of augmented reality.

[0080] As used herein, a "container" is an isolated runtime environment, such as an isolated user space instance. For example, a container can be an operating system-level virtualization unit, in which the kernel of the operating system allows and supports the existence of multiple isolated virtual runtime environments in parallel. For example, one of the following currently available container platforms can be used to implement containers: LXC, Solaris containers, Docker containers, virtual private servers (OpenVZ), virtual environments (VE), virtual kernels (DragonFlyBSD) or jails (jail) (FreeBSD jail or chroot jail). Each container can generate a runtime environment that looks like a software program running in it like a real computer.

[0081] According to some embodiments, each container is stored in or assigned to a corresponding node of a graph of a graph database of a database system described herein. Each container includes a software program and / or function, which is configured to monitor and / or control the position, orientation and / or action of one or more physical objects (e.g., raw materials, equipment, etc.) represented by the node, rather than the position, orientation and / or action of the physical objects represented by other nodes of the graph. Thus, each container can encapsulate functions for monitoring and / or controlling the one or more objects. This can allow for preventing unforeseen errors and conflicts that may be caused by unknown software interdependencies. For example, when a device driver of a machine is replaced by a newer version and the control of another machine depends on an outdated version of the driver, software problems may occur.

[0082] Depending on the embodiment, the degree to which containers are isolated from one another may vary to allow at least limited, well-defined, and therefore highly secure exchange of data between the containers and the software included therein.

[0083] According to some embodiments, each container implements an application programming interface (API) that enables the container to exchange data exclusively with other containers via the API. This can prevent situations where software dependencies between multiple devices are supervised, and can allow container-individual data exchange and control protocols to define which other containers can exchange what type of data with the container via the API.

[0084] According to other embodiments, there may be a central data exchange interface provided by the container management software at the application level. Thus, exchanges between containers are achieved in a centralized and less formal approach, which can still prevent errors caused by hidden software interdependencies.

[0085] A computer program running on a normal operating system can "see" (detect the existence of and / or access) all of the resources of the computer (connected devices, files and folders, network shares, CPU power, quantifiable hardware capabilities). However, according to some embodiments of the present invention, a program running inside a container can only "see" the contents of the container and the devices assigned to the container. In this case, the software in the container can only see the objects represented by the graph nodes assigned to the container. In some embodiments, containers are programmed and / or configured so that they can exchange data with other containers only via one or more APIs provided by the corresponding container and / or central container management software (and therefore at the application layer).

[0086] For example, the one or more objects monitored and / or controlled by the software of the container may be or include one or more sensors and / or actuators, equipment or materials of equipment (e.g., tools, machines, robots). Thus, the software within the container can only "see" and optionally modify the parameters of the equipment represented by the corresponding node of the container. It cannot see or manipulate objects represented by other containers assigned to other nodes unless there is a defined API for exchanging data with the other container.

[0087] According to some embodiments, each of the containers includes software for monitoring and / or operating one or more objects represented by the node, and includes a location reporting module. The location reporting module is a software program or function configured to monitor the location of at least one of the one or more objects represented by the corresponding node and update the spatial information associated with at least one object of the corresponding node in the database system. For example, a node may represent an object that may include a GPS sensor or other components for repeatedly and automatically determining its own position. The location information is automatically transmitted from the location sensor to the location reporting module and stored in the database system by the location reporting module. This can ensure that any changes in the location of the equipment (e.g., the current location of the movable robot) are automatically propagated to the location information stored in the database system. This allows automatic determination of whether the equipment that has moved in the direction of the user with augmented reality glasses is now so close to the user wearing AR glasses that data related to the object will be displayed via the AR glasses. For example, the location reporting module may be a software program that has subscribed to a device-specific streaming service provided by the database system. Whenever the corresponding equipment has actively or passively changed position, sensors on the equipment and / or in the environment detect the change in position of the equipment and the new position of the equipment, and update the location information of the equipment in the database system accordingly. Updating the location information in the database triggers submission of a message from the database system to a location reporting module of a container assigned to a node representing the equipment. The program logic of the augmented reality glasses is configured to use the updated location information of the equipment to check whether the distance between the user wearing the AR glasses and the equipment is small enough to display the data assigned to the equipment.

[0088] According to some embodiments, one or more of the nodes of the graph may include two or more containers or be stored in association with two or more containers. Each of the containers may include software for monitoring and / or operating one or more physical objects. This may be beneficial because the topology of the graph and the topology of the containers and the selection of objects monitored and / or controlled by the corresponding containers can be flexibly adapted to the specific needs of a given industrial manufacturing facility.

[0089] Using containers can have the following advantages: the activities of the container will not have any hidden or unforeseen effects on the activities of other containers. The graph nodes representing the container can be freely deleted, added or moved within the graph without affecting the activities in other containers. Similarly, the software contained in the container can be changed or replaced without affecting the software for monitoring and / or controlling other objects in an unforeseen way. In addition, objects can be moved in the real world without the risk that the object state or position may interfere with the software for monitoring and / or controlling physical objects represented by other containers. The changed position of the object will be identified by the position reporting module of the corresponding container, but this will not affect the data and software stored in other containers. This can reduce the downtime of the AR system because the individual nodes and corresponding hardware objects can be easily modified or replaced without the risk of hindering the normal operation of other hardware objects and their software programs. Since the container is configured as a standardized software unit, or in other words, it is configured as an application package including all its libraries and dependencies, the container can be executed (i.e., run) quickly and reliably regardless of the architecture of the database system.

[0090] In another embodiment, each container in the container advantageously and automatically responds to the movement and / or rearrangement of the object in the factory and updates the configuration of the factory. For example, the configuration can be a file in a database or a set of files or a set of one or more data records. The configuration can represent the overall equipment (machine, tool, robot, etc.) of the factory that can be used to perform the production process, and includes an indication of the position of each equipment in the equipment. Embodiments of the present invention can have the advantage that the configuration is always and fully automatically kept up to date. When the operator transports a specific tool, machine or machine part from one production unit to the next production unit, the configuration of the factory is automatically updated because the positioning system will automatically determine the relocation of the tool, machine or machine part and automatically update the configuration of the factory.

[0091] In the case where the above-mentioned nodes should represent multiple physical objects (including objects with GPS sensors and other objects without positioning sensors), the location information of the object with the GBS sensor can be used to at least approximately monitor the positions of other physically coupled objects, even in the case where these other physical objects do not include position determination sensors.

[0092] According to an embodiment, the spatial information associated with the object of the corresponding node includes one or more spatial coordinates of the object. The one or more spatial coordinates correspond to one or more points on and / or within the boundary surface of the object.

[0093] As used herein, a "bounding surface of an object" is the surface of the smallest possible 3D wrapper object that completely surrounds the entire object. Typically, a 3D wrapper object is a rectangular box. Using the bounding surface of an object instead of a real and usually much more topologically complex surface to perform spatial operations (distance determination, etc.) in a database system can have the advantage that the speed of spatial operations can be greatly increased. Due to the limited complexity of the wrapper object (particularly a cube or rectangular box), geometric operations such as distance determination, calculation of intersection areas, intersection volumes, connected areas, connected volumes, etc. can be performed with much less CPU capacity than performing these operations on a "real" object surface representation.

[0094] A "boundary space" is a space near a wearer of AR glasses in which virtual objects may be displayed to the wearer. For example, a bounding space may or may not surround the user.

[0095] According to an embodiment, the bounding space determines whether to render a virtual object, whereby the virtual object is rendered if and only if the virtual object is within the bounding space. For example, the rendering may include transforming a 2D or 3D model of an object into a pixel representation of at least a portion of the object. Additionally or alternatively, the rendering may include receiving data associated with the virtual object from one or more data sources and processing the received data so that at least a portion of the data is transformed into a pixel representation of the data. Typically, rendering is a CPU-consuming process, and retrieving data from a database may also include retrieving data via a network connection, and therefore may generate a large amount of network traffic.

[0096] A "bounding surface of a bounding space" is defined as a surface that defines (ie, encloses) the bounding space.

[0097] In another aspect, a method of using an automation system for controlling an automated production process of an industrial plant is provided. The industrial plant includes a plurality of equipment for performing the production process. The plurality of equipment is spatially distributed in the industrial plant. The method includes:

[0098] -Providing an automation system, the automation system comprising an augmented reality system, the augmented reality system comprising:

[0099] o a database system having stored data and spatial coordinates associated with respective equipment of the plurality of equipment;

[0100] o Augmented reality glasses configured to display data; and

[0101] o executable program logic coupled to the augmented reality glasses,

[0102] - receiving, by the executable program logic, the latest spatial position of the user wearing the augmented reality glasses from a positioning system,

[0103] - data of equipment located near the user's most recent spatial location as determined by the database system,

[0104] - receiving, by the executable program logic, data of the determined equipment from the database system;

[0105] -The augmented reality glasses are controlled by the executable program logic to display at least some of the received data, the displayed data enabling the user to control and / or maintain the automated production process.

[0106] Additional embodiments of the present disclosure include: Clause 1: An automation system for controlling an automated production process of an industrial plant, the industrial plant including a plurality of equipment for executing the production process, the plurality of equipment being spatially distributed in the industrial plant, the automation system including an augmented reality system. The augmented reality system includes: a database system configured to store data and spatial coordinates associated with corresponding equipment among the plurality of equipment; augmented reality glasses configured to display the data; and executable program logic coupled to the augmented reality glasses, the executable program logic configured to receive the latest spatial position of a user wearing the augmented reality glasses from the positioning system 108 and the data of equipment located near the latest spatial position of the user from the database system, the executable program logic also configured to control the augmented reality glasses to display at least some of the received data. Clause 2: An automation system according to clause 1, wherein the augmented reality system also includes the positioning system, which is configured to sense the spatial position of the user in the plant. Clause 3: The automation system of clause 2, wherein the positioning system comprises a spatial mapping grid covering at least a portion of the plant, an indoor positioning system, a cellular positioning system, or a GPS receiver configured to determine the spatial position. Clause 4: The automation system of clause 3, wherein the spatial mapping grid comprises spatial anchors, wherein the spatial anchors are interconnected, and wherein each spatial anchor is associated with a coordinate system associated with the augmented reality glasses used to display the data and a spatial coordinate system associated with the spatial coordinates of one or more of the plurality of equipment stored in the database system, wherein the program logic is configured to display the data of the equipment located near the user's most recent spatial position at a predefined position relative to the spatial anchor via the augmented reality glasses. Clause 5: An automation system according to any of the preceding clauses, wherein the augmented reality system is configured to create an avatar for at least one remote user, the remote user being remote from the industrial plant, wherein the program logic is configured to display the avatar via the augmented reality glasses near the latest spatial position of the user wearing the augmented reality glasses, wherein preferably the augmented reality system is configured to provide a two-way visual and / or acoustic communication channel between the user wearing the augmented reality glasses and the remote user. Clause 6: An automation system according to clause 5, wherein the augmented reality glasses include an acoustic output interface, wherein the augmented reality system is configured to provide a two-way acoustic communication channel between the user wearing the augmented reality glasses and the avatar, and wherein the program logic is configured to control the acoustic output interface such that the volume of the avatar's voice output to the user wearing the augmented reality glasses via the acoustic output interface is negatively correlated to the distance between the user wearing the glasses and the avatar within the coordinate system associated with the augmented reality glasses used to display the data.Clause 7: An automation system according to clause 5 or 6, wherein the automation system includes a robot having a camera and / or a microphone, the robot being located in the industrial plant, wherein the augmented reality system includes a robot control module, the robot control module being configured to control the movement and / or orientation of the robot so that the position and / or the orientation of the robot reflects the position and / or the orientation of the avatar within the augmented reality displayed via the AR glasses, and wherein the robot control module is configured to automatically update the position and / or the orientation of the avatar and the robot according to changes in the position and / or the orientation of the remote user or in response to a control command submitted by the remote user. Clause 8: An automation system according to any of the preceding clauses, wherein the database system is a graph database system, a spatial database system, and / or a streaming database system. Clause 9: The automation system according to any of the preceding clauses, further comprising: a distributed control system coupled to the database system, the distributed control system comprising: a memory configured to store one or more process parameters associated with one or more of the equipment and store process control software; and a processor configured to execute the process control software for automatically controlling the production process. Clause 10: The automation system according to clause 9, further comprising: one or more sensors coupled to at least one equipment, the one or more sensors configured to measure the one or more process parameters of the at least one equipment, the one or more process parameters indicating the current state of the production process of the factory and / or the current operating state or mode of the equipment; and one or more actuators coupled to the at least one equipment, the one or more actuators configured to control the one or more process parameters or configured to operate according to the one or more control parameters, wherein the distributed control system is coupled to the one or more sensors and the one or more actuators, wherein the processor is configured to execute the process control software for automatically controlling the production process based on measurement signals received from the one or more sensors and control signals sent to the one or more actuators. Clause 11: An automation system according to any of the preceding clauses, the automation system further comprising a user database coupled to the distributed control system, the user database being configured to store user information associated with the production process, wherein the user information comprises a user ID, a user role and / or user permissions.Clause 12: An automation system according to clause 11, wherein the executable program logic is configured to receive user credentials from the user and authenticate the user, wherein the user ID of the user wearing the augmented reality glasses is associated with one or more of the user roles and each of the one or more user roles includes one or more of the user permissions, and wherein the executable program logic is also configured to select and display the content of at least some of the received data for display based on the one or more user roles and / or the one or more user permissions associated with the user ID. Clause 13: An automation system according to clauses 11 or 12, wherein the one or more user roles include one or more of a maintenance role and an engineering design role, and the one or more user permissions define access to one or more specific equipment of the plurality of equipment. Clause 14: An automation system according to any of the preceding clauses, wherein the data includes one or more task instructions associated with the corresponding equipment. Clause 15: An automation system according to any of the preceding clauses, wherein one or more of the plurality of equipment is one or more machines. Clause 16: An automated system according to any of the preceding clauses, wherein the executable program logic is configured to perform a query on the database based on the latest spatial location to retrieve from the database the data of the equipment located near the latest spatial location of the user. Clause 17: An automated system according to any of the preceding clauses, wherein the database system is accessible via a subscription-based streaming service, and wherein the executable program logic is configured to receive from the database via the streaming service the data of the equipment located near the latest spatial location of the user. Clause 18: An automated system according to clause 16 or 17, wherein the database system is configured to selectively provide the data of equipment located near the latest spatial position of the user wearing the augmented reality glasses in response to the query of the executable program logic according to clause 16, whereby the movement of the user relative to the equipment triggers the executable program logic to submit a new query to the database system including the new latest spatial position of the user; or wherein the database system is configured to selectively provide the data of equipment located near the latest spatial position of the user wearing the augmented reality glasses via a subscription-based streaming service according to clause 17, whereby the movement of the user relative to the equipment triggers the executable program logic to update the subscription with the new latest spatial position of the user.Clause 19: An automation system according to any of the preceding clauses, wherein the executable program logic is configured to: receive coordinates of a visual object that graphically represents the data of the equipment near the user wearing the augmented reality glasses; determine the line of sight of the user in a real-world coordinate system; receive coordinates of a real-world object near the latest spatial position of the user wearing the augmented reality glasses from the database system; determine whether one or more of the virtual objects representing the data of the equipment are located along the line of sight of the user and are farther from the user than at least one of the real-world objects that are also located along the line of sight of the user; and non-selectively display the one or more virtual objects that are determined to be on the line of sight of the user and located farther than at least one of the real-world objects. Clause 20: An automation system according to any of the preceding clauses, wherein the database system is a graph database system, the graph database system comprising a graph database having a plurality of interconnected nodes, wherein each node represents one or more objects of the industrial plant and comprises spatial information associated with the one or more objects, and wherein the one or more objects are selected from the group consisting of sensors, actuators, raw materials and / or one of the plurality of equipment, the equipment being, for example, a machine or a robot. Clause 21: An automated system according to clause 20, wherein one or more of the nodes include at least one corresponding container or are stored in association with the at least one corresponding container, each container being an isolated runtime environment instance, the runtime environment instance including software for monitoring and / or operating at least one of the one or more objects represented by the node, and wherein the software in each container includes a location reporting module, the location reporting module being configured to monitor the location of at least one of the one or more objects of the corresponding node and update the spatial information associated with the at least object of the corresponding node in the database system. Clause 22: An automated system according to clause 21, wherein the spatial information associated with the at least one object of the corresponding node includes one or more spatial coordinates of the at least one object, the one or more spatial coordinates corresponding to one or more points on and / or within the boundary surface of the object. Clause 23: An automated system according to any of the preceding clauses, the automated system also including a network that communicatively couples the database system to the augmented reality glasses.Item 24: A method of using an automation system to control an automated production process of an industrial plant, the industrial plant including a plurality of equipment for executing the production process, the plurality of equipment being spatially distributed in the industrial plant, the method comprising: providing the automation system, the automation system including an augmented reality system, the augmented reality system including: a database system storing data and spatial coordinates associated with corresponding equipment among the plurality of equipment; augmented reality glasses configured to display the data; and executable program logic coupled to the augmented reality glasses; the executable program logic receiving the latest spatial position of a user wearing the augmented reality glasses from a positioning system; the database system determining the data of equipment located near the latest spatial position of the user; the executable program logic receiving the data of the determined equipment from the database system; the executable program logic controlling the augmented reality glasses to display at least some of the received data, the displayed data enabling the user to control and / or maintain the automated production process.

[0107] These embodiments may be combined with any of the hands-free embodiments described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0108] The following embodiments of the invention are explained in more detail, by way of example only, with reference to the accompanying drawings, in which:

[0109] Figure 1A An automated system for controlling an automated production process of an industrial plant according to an embodiment of the present invention is shown;

[0110] Figure 1B shows a top view of AR glasses worn on a user's head according to one embodiment of the present invention;

[0111] Figure 1C shows a top view of AR glasses worn on a user's head according to another embodiment of the present invention;

[0112] Figure 1D A top view of AR glasses worn on a user's head according to yet another embodiment of the present invention is shown;

[0113] Figure 1E shows the bounding space around a user wearing AR glasses;

[0114] Figure 1F A bounding space close to but not surrounding a user wearing AR glasses is shown;

[0115] Figure 1G Two different bounding spaces for two different types of virtual objects are shown;

[0116] Figure 1H A bounding space is shown whose position and / or orientation dynamically adapts to the gaze of a user wearing the AR glasses.

[0117] Figure 2 An exemplary task instruction according to one embodiment of the present invention is shown;

[0118] Figure 3 An exemplary node of a graph database is shown, the node representing a task instruction;

[0119] Figure 4 A flow chart showing a method for operating an automation system;

[0120] Figure 5 An exemplary GUI displayed to an "Operator" user is shown; and

[0121] Figure 6 An exemplary GUI displayed to a "Maintenance Worker" user is shown;

[0122] Figure 7 shows the use of a spatial anchor and shows an avatar of a remote user;

[0123] Figure 8 A coordinate system for representing real-world equipment and a local user avatar in a virtual metaverse to a teleoperator is shown;

[0124] Fig. 9 A flow chart showing a method for operating an automation system according to another embodiment of the present disclosure is shown;

[0125] Fig.10 shows a portion of an industrial plant with equipment located close to a user;

[0126] Fig.11 A flow chart showing a method for controlling an automation system according to yet another embodiment of the present disclosure is shown;

[0127] Fig.12 shows a list of processes displayed to a user of the AR glasses via enhanced AR glasses according to one embodiment of the present disclosure;

[0128] Fig.13 showing a first task displayed to the user via the AR glasses according to an embodiment of the present disclosure;

[0129] Fig.14 showing a next task displayed to the user of the AR glasses according to one embodiment of the present disclosure; and

[0130] Fig.15A flow chart of a method for controlling an automation system according to an embodiment of the present disclosure is shown, the automation system being used to control an automated production process of an industrial factory. DETAILED DESCRIPTION

[0131] Descriptions of various embodiments of the present invention have been given for illustrative purposes, and these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and substance of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications or technical improvements over technologies available on the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

[0132] "Industrial plant" or "factory" may refer to, but is not limited to, any technical infrastructure for industrial purposes (i.e., a manufacturing or production process or processing performed by an industrial plant) for manufacturing, producing or processing one or more industrial products. Industrial products may be, for example, any physical product, such as chemicals, biological products, medicines, foods, beverages, textiles, metals, plastics, semiconductors. Additionally or alternatively, chemical products may even be service products, such as recycling or waste treatment (such as recycling), chemical treatment (such as decomposition or dissolution into one or more chemical products). Therefore, an industrial plant may be one or more of a chemical plant, a processing plant, a pharmaceutical plant, a fossil fuel processing facility (such as an oil well and / or a natural gas well), an oil refinery, a petrochemical plant, a cracking plant, etc. An industrial plant may even be any one of a distillation plant, a treatment plant, or a recycling plant. An industrial plant may even be any one of the examples given above or a combination of their analogs.

[0133] The infrastructure may include equipment or process units, such as any one or more of the following: heat exchangers, towers (such as distillation towers), furnaces, reaction chambers, cracking units, storage tanks, extruders, granulators, precipitators, agitators, mixers, cutters, curing tubes, evaporators, filters, screens, pipes, chimneys, filters, valves, actuators, mills, transformers, conveying systems, circuit breakers, machinery (e.g., heavy rotating equipment such as turbines, generators, pulverizers, compressors, industrial fans, pumps), conveying elements (such as conveyor systems), motors, etc.

[0134] In addition, industrial plants typically include a plurality of sensors and at least one control system for controlling at least one parameter associated with a process or process parameter in the plant. Such control functions are typically performed by a control system or controller in response to at least one measurement signal from at least one of the sensors. The controller or control system of the plant may be implemented as a distributed control system ("DCS") and / or a programmable logic controller ("PLC").

[0135] Therefore, at least some of the equipment or process units of the industrial plant can be monitored and / or controlled to produce one or more industrial products in the industrial product. Even monitoring and / or control can be performed to optimize the production of one or more products. The equipment or process unit can be monitored and / or controlled via a controller (such as a DCS) in response to one or more signals from one or more sensors. In addition, the factory can even include at least one programmable logic controller ("PLC") for controlling some processes in the process. Industrial plants can generally include a plurality of sensors that can be distributed in the industrial plant for monitoring and / or control purposes. Such sensors can generate a large amount of data. The sensor may or may not be considered as part of the equipment. Therefore, production such as chemical and / or service production may be a data-intensive environment. Therefore, each industrial plant may generate a large amount of process-related data. Those skilled in the art will understand that industrial plants generally may include instruments that may include different types of sensors. Sensors can be used to measure one or more process parameters and / or for measuring equipment operating conditions or parameters related to equipment or process units. For example, sensors may be used to measure process parameters such as flow rate in a pipe, liquid level in a tank, temperature of a furnace, chemical composition of a gas, etc., and some sensors may be used to measure vibration of a pulverizer, speed of a fan, opening of a valve, corrosion of a pipe, voltage across a transformer, etc. The differences between these sensors are not only based on the parameters they sense, but even the sensing principle used by the respective sensors. Some examples of sensors based on the parameters sensed by the sensor may include: temperature sensors, pressure sensors, radiation sensors such as light sensors, flow sensors, vibration sensors, displacement sensors, and chemical sensors such as for detecting specific substances such as gases. Examples of sensors that differ in the sensing principle employed may be, for example: piezoelectric sensors, piezoresistive sensors, thermocouples, impedance sensors such as capacitive sensors and resistive sensors, etc.

[0136] Industrial plants can even be part of multiple industrial plants. As used herein, the term "multiple industrial plants" is a broad term, and is given its common and customary meaning for those of ordinary skill in the art, and is not limited to special or custom meanings. The term specifically may refer to, but is not limited to, a complex of at least two industrial plants with at least one common industrial purpose. Specifically, multiple industrial plants may include at least two, at least five, at least ten or even more industrial plants that are physically and / or chemically coupled. Multiple industrial plants can be coupled so that the industrial plants forming multiple industrial plants can share one or more of their value chains, educts and / or products. Multiple industrial plants may also be referred to as complexes, composite sites, "integration" or "integration sites". In addition, the value chain production of multiple industrial plants from various intermediate products to final products can be dispersed in various locations, such as in various industrial plants, or integrated in integrated sites or chemical parks. Such integrated sites or chemical parks may be or may include one or more industrial plants, wherein the product manufactured in at least one industrial plant may be used as the raw material of another industrial plant.

[0137] As used herein, "automation system" refers to a system used to automatically control processes, such as, for example, chemical plants, refineries, paper mills, and pulp mills. An automation system may use a network to interconnect sensors, controllers, operator terminals, and actuators. It may be based on open standards or proprietary standards. Process automation can be used to operate power plants and factories in different types of industries more efficiently and safely.

[0138] A "production process" is any industrial process that, when used on or applied to input materials, provides a product. The product may be, for example, a mechanical product, an electronic product and / or a chemical product. Thus, a production process may be any manufacturing or processing process or a combination of processes for obtaining a chemical product. A production process may even include packaging and / or stacking of the product.

[0139] The terms "manufacturing," "producing," or "processing" will be used interchangeably in the context of a production process. The term can include any type of application of an industrial process to input materials that produces one or more products.

[0140] "Chemical product" in the present disclosure may refer to any industrial product, such as chemical product, pharmaceutical product, nutritional product, cosmetic product or biological product, or even any combination of their combinations. Chemical product may be composed entirely of natural ingredients, or it may at least partially include one or more synthetic ingredients. Some non-limiting examples of chemical products are organic or inorganic compositions, monomers, polymers, foams, pesticides, herbicides, fertilizers, feeds, nutritional products, precursors, drugs or therapeutic products, or any one or more of their ingredients or active components. In some cases, chemical products may even be products that can be used by end users or consumers, such as cosmetics or pharmaceutical compositions. Chemical products may even be products that can be used to further manufacture one or more products, such as synthetic foams that can be used to manufacture soles or coatings that can be used for the exterior of automobiles. Chemical products may be in any form, such as solid, semi-solid, pasty, liquid, emulsion, solution, granular, particle or powder.

[0141] Production processes can be continuous or cyclical, for example, a batch chemical production process when using catalysts that need to be recycled. One major difference between these production types is the frequency of occurrence in the data generated during production. For example, in a batch process, the production data extends from the beginning of the production process to the last batch of different batches that have been produced in that run. In a continuous setting, the data is more continuous with potential changes in production operations and / or maintenance-driven downtime.

[0142] "Process parameter" may refer to any one of the variables associated with the production process, such as any one or more of temperature, pressure, time, liquid level, etc. A process parameter may indicate the current state of equipment, equipment parts, materials, or products. According to some embodiments, one or more of the process parameters indicate the operating mode of the equipment or equipment parts. The process parameters may be measured by sensors. Other process parameters may be desired parameter values ​​or value ranges, and may be set and / or dynamically modified by a user. According to an example, a process parameter may be a numerical value or a binary signal value measured, for example, via one or more sensors during the production process. Preferably, the process parameters are received and stored in association with time information indicating the time when the process parameters were measured.

[0143] "Input material" or "raw material" may refer to at least one raw material or unprocessed material used to produce a chemical product. An input material may be any organic or inorganic substance or even a combination thereof. Thus, an input material may even be a mixture, or it may include a variety of organic and / or inorganic components in any form. In some cases, an input material may even be an intermediate processed material, for example from an upstream processing facility or plant.

[0144] As used herein, a "spatial database system" is a software application designed to allow the definition, creation, querying, updating, and management of spatial databases.

[0145] As used herein, a "spatial database" is a database that supports database operations that are optimized for storing and querying data representing objects defined in a geometric space. A spatial database system allows the representation of simple geometric objects such as points, lines, and polygons, and optionally also represents more complex structures such as 3D objects, topological overlays, and / or linear networks. Although typical databases are designed to manage data of various digital types and character types, spatial databases include additional functionality for efficiently processing spatial data objects. Spatial data objects may also be referred to as "spatial primitives" or "simple geometric objects." The term "geometry" may be used to refer to individual spatial data objects and to aggregates of spatial data objects.

[0146] As used herein, a "spatial database operation" is a database routine configured to analyze spatial information of one or more spatial data objects. Spatial database operations are typically optimized for speed to efficiently process spatial data objects by using spatial indexes created for multiple spatial data objects. For example, a spatial database may support one or more of the following spatial database operations: spatial measurements (e.g., calculating line lengths, areas of polygons, distances between spatial data objects), spatial functions (modifying existing spatial data objects to create new spatial data objects, such as by providing a minimum bounding rectangle around them, intersecting spatial data objects, merging spatial data objects); spatial predicates (performing true / false queries about spatial relationships between spatial data objects, for checking, for example, whether polygons overlap or are within a maximum distance from another spatial data object), and the like.

[0147] As used herein, a "streaming database system" is a database system accessible via a subscription-based streaming service. The data inside a streaming database system is usually divided into two layers. The original input, usually called a "stream", is an immutable sequence of append-only events. They refer to the historical record of what happened and when it happened. The second layer is built by watching the stream and building a summary (usually statistical) about the event. For example, they can count the number of times an event occurred every day in the last month, or find the average value for each week of a year. The analysis is usually stored in a table whose structure and behavior are usually similar to traditional relational databases. In fact, according to the embodiment, the streaming service is associated with and / or based on a traditional database for these results. Using a streaming database can have the following advantages: the size of the data can be reduced, and storage costs can be saved. For example, parameter values ​​collected by equipped sensors every second and the average value calculated in one day can be received via a streaming interface. Storing only the statistically aggregated parameter values ​​of at least some of the process parameters in the process parameters can make long-term tracking technically feasible.

[0148] In another advantageous aspect, the streaming database makes it easier for developers to integrate parameter values ​​received via one or more streams, because the streams are directly accessible in the streaming database system, compared to traditional databases where the sequence of insert or delete events is hidden in an internal log or ledger. Developers can adjust how to statistically aggregate and store the streams of new parameter values ​​in a tabular form. In addition, the streaming database system supports subscription-based data retrieval, such as for dynamically determining which equipment is spatially close to the user wearing AR glasses, whereby the equipment and / or the user can change position. The user can subscribe to the service using his current location as an independent variable or condition. The service will continuously and automatically determine whether the equipment is spatially close to the user, and will enable the executable program logic of the AR glasses to selectively generate and display a GUI with equipment data of equipment that is spatially close to the user. Each movement of the user wearing AR glasses can automatically trigger an update of the subscribed streaming service using the user's new location information. For example, Oracle Stream, Apache Kafka, or IBM's "Stream" can be used as a streaming database system.

[0149] As used herein, the expression "graph database system" is a database system that uses a graph structure for semantic queries with nodes, edges and attributes to represent and store data. The key concept of the graph database system is a graph (or edge or relationship). The graph associates data items in the repository (e.g., equipment and associated mission data records, location information, setting parameters or measurement parameters) with a collection of nodes and edges, and the edge represents the relationship between the nodes. The relationship allows the data in the repository to be directly linked together and retrieved with one operation in many cases. The graph database will keep the relationship between the data as a priority. Querying relationships is fast because the relationships are permanently stored in the database. The graph database can be used to intuitively visualize relationships, making them useful for heavily interconnected data. For example, according to an embodiment, the graph represents equipment (e.g., machines, tools and / or robots), equipment components and subcomponents thereof in multiple different hierarchical layers. Once a user approaches a specific equipment and approaches the equipment spatially, the graph database system can be used to quickly retrieve not only the data of a single node representing the specific equipment in a single query operation, but also the data of any equipment component or subcomponent (direct child node or second-order child node, third-order child node or n-order child node). Thus, once the user is spatially close to the equipment, the executable program logic will receive all relevant parameters of the equipment and its subcomponents and can immediately display the parameters via the AR glasses, or some of the retrieved data can be cashed out for later display without further queries to the database system. Using a graph database can be particularly advantageous for retrieving in-depth analysis of data related to a specific maintenance task for one or more of the machines and their components.

[0150] As used herein, the expression "augmented reality glasses" or "AR glasses" refers to wearable computer glasses that add information next to or to what the wearer sees. Superimposing information onto the field of view can be achieved, for example, by an optical head-mounted display (OHMD) or embedded wireless glasses with an augmented reality overlay. These systems have the ability to reflect a projected digital image and allow the user to see through it or see better with it. Some models include a processor with limited processing power, and it only performs basic tasks, such as serving as a front-end display for a remote system with a larger amount of CPU capacity, as in the case of smart glasses utilizing cellular technology or Wi-Fi. Modern AR glasses are effective wearable computers that can run independent mobile applications. According to a preferred embodiment, the AR glasses are hands-free and can communicate with one or more external software programs (e.g., executable program logic of the AR system) via natural language voice commands and / or optical sensors combined with gesture recognition capabilities.

[0151] According to an embodiment, the AR glasses may collect information from internal sensors or external sensors, for example, to determine the position and / or orientation of the wearer, and may be configured to represent part of a positioning system of the AR system. According to an embodiment, the AR glasses support wireless technologies such as Bluetooth and / or Wi-Fi for data exchange with other components of the AR system and / or GPS for location determination. The AR glasses may include an acoustic output interface configured to output sounds (e.g., voices of other users) and / or alert messages generated by the AR system (e.g., by executable program logic).

[0152] As used herein, the expression "positioning system" is a monolithic technology system or a distributed technology system configured to determine the location of an entity (e.g., a user or another physical object). For example, some AR glasses include a GPS sensor or an indoor location sensor, a gyroscope, and / or a camera that is configured to continuously acquire data that allows the current location of the user wearing the AR glasses to be determined.

[0153] The expression "spatial anchor" as used herein is a software function that allows virtual reality objects such as holograms, virtual GUIs, or manuals to be positioned at a defined distance relative to the anchor. An anchor represents a coordinate system that is mapped and bound to a coordinate system of the real world. According to some embodiments, the anchor is created by or with the help of an executable program logic (e.g., an augmented reality application) that can interoperate with AR glasses. These anchors are then stored in a database system so that they can be accessed by other users (e.g., remote users). A virtual reality application configured to create virtual reality for a remote user can be configured to read an anchor created by a user wearing AR glasses, and to create and display some virtual objects at a defined distance and orientation relative to the anchor. The augmented reality application of a user wearing AR glasses can do the same thing. Thus, a spatial anchor allows two users to share the same visual experience. For example, a virtual object displayed at a predefined distance and orientation of an anchor can be a virtual representation of an equipment or equipment part or a virtual representation of a GUI designed to allow a user to control equipment. According to one example, a spatial anchor can be implemented using Microsoft Azure technology.

[0154] The term "avatar" as used herein is a graphical representation of a user or a persona or character of a user. When displayed via AR glasses, it can take a two-dimensional form or a three-dimensional form.

[0155] The expression "near" or "close to" or "near the user" of a user (or other object / entity) may mean, for example, within a user-defined or predefined maximum distance from the user (or from other object / entity). For example, the maximum distance may be less than 200m, or less than 10m, or less than 5m, or less than 2m, or less than 1m. The maximum distance may also be specified in the form of a 3D object (e.g., a hemisphere or cylinder), or in the form of a 2D object (e.g., a circle, an ellipse, or a polygon), in some cases with a major and / or minor axis defined, so that the distance thresholds for "neighboring" and "non-neighboring" objects may be different for different dimensions within a real-world 3D coordinate system.

[0156] According to some embodiments, a "boundary space near a user" is a boundary space surrounding a user. According to some embodiments, a "boundary space near a user" is a boundary space that does not surround a user but is close to the user. The maximum distance of a boundary space "near" a user can vary according to the embodiment. According to some examples, a boundary space "near" a user is a boundary space that is located less than 200m, or less than 10m, or less than 5m, or less than 2m, or less than 1m from the user at the point farthest from the user.

[0157] For example, in some examples, the bounding space may be a bounding space that includes the user and has a radius in the range of 2m to 200m. This may be advantageous, for example, for alarm messages that may be associated with machines located in different rooms or floors. In some further examples, the maximum distance of a bounding space that is "close" to the user may be, for example, a bounding space whose farthest point from the user is less than 10m (e.g., less than 5m from the user). This maximum distance may be useful, for example, for virtual objects that include video and text.

[0158] A "computing unit" may include or may be a processing member or computer processor having one or more processing cores, such as a microprocessor, a microcontroller, etc. In some cases, the computing unit may be at least partially part of an equipment, for example it may be a process controller such as a programmable logic controller ("PLC") or a distributed control system ("DCS"), and / or it may be at least partially a remote server. Thus, the computing unit may receive one or more input signals from one or more sensors operatively connected to the equipment. If the computing unit is not part of the equipment, it may receive one or more input signals from the equipment. Alternatively or in addition, the computing unit may control one or more actuators or switches operatively coupled to the equipment. The one or more actuators or switches may even be part of the equipment operatively.

[0159] "Memory storage device" may refer to a device for storing information in the form of data in a suitable storage medium. Preferably, the memory storage device is a digital storage device suitable for storing information in a machine-readable digital form (e.g., digital data readable via a computer processor). The memory may be a transient or non-transient storage device for information. Thus, the memory storage device may be implemented as a digital memory storage device readable by a computer processor. Further preferably, the memory storage device on the digital memory storage device may also be manipulated via a computer processor. For example, any portion of the data recorded on the digital memory storage device may be partially or completely written and / or erased and / or rewritten by a computer processor with new data.

[0160] Thus, the computing unit may be capable of manipulating one or more parameters related to the production process by controlling any one or more of the actuators or switches and / or the end effector unit, for example via manipulating one or more of the equipment operating conditions. Preferably, the control is performed in response to one or more signals retrieved from the equipment.

[0161] An "interface" may be a hardware and / or software component, or at least partially a part of an equipment, or a part of another computing unit that provides an object identifier. In some cases, an interface may also be connected to at least one network, for example, for interfacing two hardware components and / or protocol layers in a network. For example, an interface may be an interface between an equipment and a computing unit. In some cases, the equipment may be communicatively coupled to the computing unit via a network. Thus, an interface may even be a network interface, or it may include a network interface. In some cases, an interface may even be a connection interface, or it may include a connection interface.

[0162] "Network interface" refers to a device or set of one or more hardware and / or software components that allows an operational connection to a network.

[0163] "Connection interface" refers to a software and / or hardware interface for establishing communication, such as transmitting or exchanging signals or data. The communication may be wired or wireless. The connection interface is preferably based on or supports one or more communication protocols. The communication protocol may be a wireless protocol, such as: The connection interface may be based on a short-range communication protocol such as Wi-Fi or WiFi, or a long-range communication protocol such as a cellular or mobile network, such as a second generation cellular network or ("2G"), 3G, 4G, Long Term Evolution ("LTE"), or 5G. Alternatively or in addition, the connection interface may even be based on a proprietary short-range or long-range protocol. The connection interface may support any one or more standard and / or proprietary protocols. The connection interface and the network interface may be the same unit, or they may be different units.

[0164] The "network" discussed herein can be any suitable type of data transmission medium, wired, wireless or a combination thereof. A particular type of network does not limit the scope or generality of the present teachings. Thus, a network can refer to any suitable arbitrary interconnection between at least one communication endpoint and another communication endpoint. A network may include one or more distribution points, routers or other types of communication hardware. The interconnection of networks can be formed by physical hard-wiring, optical and / or wireless radio frequency methods. Specifically, a network can be or can include a physical network made entirely or partially by hard-wiring (such as a fiber optic network) or a network made entirely or partially by conductive cables or a combination thereof. A network may include, at least in part, the Internet.

[0165] "Equipment" may refer to any one or more assets within an industrial plant. As non-limiting examples, equipment may refer to any one or more or any combination of the following: a computing unit or controller (such as a programmable logic controller ("PLC") or a distributed control system ("DCS")), a sensor, an actuator, an end effector unit, a conveying element (such as a conveyor system), a heat exchanger (such as a heater, a furnace, a cooling unit, a reactor, a mixer, a milling machine, a shredder, a compressor, a slicer, an extruder, a dryer, a sprayer, a pressure or vacuum chamber, a pipe, a silo, and any other kind of device used directly or indirectly for production in an industrial plant or during production in an industrial plant. Preferably, equipment refers in particular to those assets, devices or components that are directly or indirectly involved in the production process. More preferably, those assets, devices or components that can affect the performance of the chemical product. Equipment can be buffered equipment or non-buffered equipment. In addition, equipment can involve mixing or non-mixing, separation or non-separation. Some non-limiting examples of non-buffered equipment without mixing are conveyor systems or conveyor belts, extruders, granulators and heat exchangers. Some non-limiting examples of buffer equipment without mixing are buffer silos, silos, etc. Some non-limiting examples of buffer equipment with mixing are silos with mixers, mixing vessels, choppers, double cone blenders, curing tubes, etc. Some non-limiting examples of non-buffered equipment with mixing are static or dynamic mixers, etc. Some non-limiting examples of buffer equipment with separation are towers, separators, extractions, thin film evaporators, filters, screens, etc. The equipment can even be or it can include storage or packaging elements, such as octagonal silo fills, drums, bags, tank trucks.

[0166] "Equipment operating condition" refers to any characteristic or value that represents the status of the equipment, for example, any one or more of set points, controller outputs, production sequences, calibration status, any equipment-related warnings, vibration measurements, speed, temperature, fouling values ​​such as filter differential pressure, maintenance dates, etc.

[0167] "Parameter" in this context refers to any relevant physical or chemical property and / or measurement thereof, such as temperature, direction, position, amount, density, weight, color, humidity, speed, acceleration, rate of change, pressure, force, distance, pH, concentration, and composition. A parameter may also refer to the presence or absence of a property.

[0168] "Actuator" refers to any component responsible for directly or indirectly moving and controlling a mechanism associated with equipment such as a machine. An actuator may be a valve, a motor, a drive, etc. An actuator may be electrically, hydraulically, pneumatically, or any combination thereof.

[0169] "Computer processor" refers to any logic circuit configured to perform the basic operation of a computer or system, and / or generally refers to a device configured to perform calculations or logical operations. In particular, a processing component or a computer processor may be configured to process the basic instructions that drive a computer or system. As an example, a processing component or a computer processor may include at least one arithmetic logic unit ("ALU"), at least one floating point unit ("FPU") (such as a mathematical coprocessor or a numerical coprocessor), a plurality of registers (specifically, registers configured to provide operands to the ALU and store operation results) and a memory (such as L1 and L2 cache memory). In particular, a processing component or a computer processor may be a multi-core processor. In particular, a processing component or a computer processor may be or may include a central processing unit ("CPU"). A processing component or a computer processor may be a complex instruction set computing ("CISC") microprocessor, a reduced instruction set computing ("RISC") microprocessor, a very long instruction word ("VLIW") microprocessor, or a processor that implements other instruction sets or a processor that implements a combination of instruction sets. The processing component may also be one or more special processing devices, such as an application specific integrated circuit ("ASIC"), a field programmable gate array ("FPGA"), a complex programmable logic device ("CPLD"), a digital signal processor ("DSP"), a network processor, etc. The methods, systems, and devices described herein may be implemented as software in a DSP, in a microcontroller, or in any other side processor, or as a hardware circuit in an ASIC, CPLD, or FPGA. It should be understood that the term processing component or processor may also refer to one or more processing devices, such as a distributed system of processing devices located on multiple computer systems (e.g., cloud computing), and is not limited to a single device unless otherwise specified.

[0170] A "computer-readable data medium" or carrier includes any suitable data storage device or computer-readable memory having stored thereon one or more sets of instructions (e.g., software) embodying any one or more of the methods or functions described herein. The instructions may also reside completely or at least partially in the main memory and / or in the processor during execution of the instructions by the computing unit, main memory, and processing device that may constitute the computer-readable storage medium. The instructions may also be transmitted or received over a network via a network interface device.

[0171] Figure 1A An automation system 100 for controlling an automated production process of an industrial plant is shown. The industrial plant comprises a plurality of equipment 122, 124 for performing the production process. In one embodiment, the plurality of equipment is spatially distributed in the industrial plant.

[0172] The automation system includes: an augmented reality system, the augmented reality system including a database system 102 configured to store data and spatial coordinates associated with corresponding equipment in a plurality of equipment; an augmented reality (AR) glasses 104 configured to display the data; and an executable program logic 106 coupled to the augmented reality glasses 104. For example, the AR glasses can be HoloLens from Microsoft, or an iOS-based device that supports, for example, ARKit, or an Android-based device that supports ARCore or a similar library. The executable program logic 106 can be integrated into the AR glasses, or can be at least partially outside the glasses and operatively coupled to the AR glasses via a wireless data exchange interface. The program logic can be used to generate a GUI and / or render a digital geometric object for display via the AR glasses. In addition, the executable program logic may include an interface for exchanging data with the database system 102 and the positioning system 108. In one embodiment, the augmented reality system includes the AR glasses 104 and the program logic 106, and the database system 102 is a component of the distributed control system 132 or an independent object or component of a factory that is communicatively coupled to the augmented reality system and / or the distributed control system 132. In one embodiment, the augmented reality system also includes a network 107 for communicatively coupling the AR glasses 104 to the database system 102. The augmented reality glasses 104 are associated with a human operator (i.e., an AR user) and include such things as electronics, optics (e.g., display, lenses, light source), power supply, local memory, processing unit, and executable software (such as AR applications known to those skilled in the art) to enable the AR glasses to provide an augmented (i.e., enhanced) vision of the factory environment as viewed by the user through the glasses. The AR glasses may also include an acoustic output interface, such as an integrated speaker and / or microphone, for enabling a user wearing the glasses to communicate with one or more remote users via voice.

[0173] In one embodiment, when the user views different objects in the factory (such as different equipment, sensors, actuators, factory structures (e.g., floors, ceilings, walls, scaffolding, storage areas, etc.) through the display of the AR glasses, the AR glasses 104 provide an overlay of data (also referred to as AR objects or "virtual objects", representing data such as text, photos, videos, holograms, GUIs, technical manuals, and / or diagrams) on the display of the AR glasses. In one embodiment, the type and content of the AR objects displayed depends on the characteristics of the user of the AR glasses, the user's location in the factory, and / or the user's line of sight, as will be described in further detail below.

[0174] In one embodiment, executable program logic 106 is configured to receive the most recent spatial location of the user from a positioning system, such as positioning system 108. In one embodiment, positioning system 108 is configured to sense the spatial location of the user in the factory.

[0175] According to one embodiment, the positioning system 108 includes one or more sensors for dynamically identifying the position and / or orientation of a user wearing AR glasses. For example, the positioning system may include a GPS module or an indoor positioning system. In addition or alternatively, the positioning system may include one or more cameras for capturing gestures of the user's hands and arms when interacting with a virtual object (such as a GUI) displayed via AR glasses. According to an embodiment, the positioning system 108 is configured to generate and / or use a spatial mapping grid covering at least a portion of the factory. For example, spatial mapping via the use of a spatial mapping grid enables the AR object to be anchored on or near the real surface of an object in the factory (such as the surface of equipment, sensors, actuators, or other surfaces in the factory, including the surface of the factory infrastructure). Embodiments of the present invention include the use of commercially available spatial mapping grid positioning systems, some of which are commercially available as applications to be executed by AR glasses 104 and / or program logic 106. One such system is World Locking Tools, available as open source code.

[0176] Additionally or alternatively, embodiments of the present invention include a pre-built spatial map (i.e., a spatial grid) defining the surfaces of objects in the factory, wherein each surface of the object has a plurality of grid points to which corresponding AR objects can be placed or locked, and by associating the coordinates of the locked grid points with the AR coordinates in the potential mobile AR coordinate system of the AR glasses, the AR objects appear to be locked in place relative to their positions with respect to the corresponding factory objects. In one embodiment, one or more spatial surfaces of the objects in the factory are represented by a triangular mesh to which the AR objects can be anchored. Thus, AR objects placed in place relative to physical world features will remain fixed relative to those features, regardless of the user's movement or the user's viewing direction through the AR glasses. For example, Microsoft's Azure spatial anchor technology can be used to develop mixed reality applications that support AR glasses from various manufacturers. Azure spatial anchors enable the development of mixed reality applications that specify precise points of interest, so-called spatial anchors, and call those points of interest from one or more supported devices so that multiple local and / or remote users can share the same virtual coordinate system and the virtual objects included therein. According to the embodiment, the spatial mapping grid includes or is connected to the spatial anchors. At least some of the spatial anchors may be interconnected and form one or more groups of interconnected spatial anchors, for example, having a predefined relative distance from each other. In one embodiment, one or more spatial anchors connected to any particular spatial anchor are defined as one or more nearest neighbor spatial anchors. In one embodiment, each spatial anchor associates the coordinate system of the augmented reality system used for display of data (e.g., AR objects) with the real-world coordinate system. In other words, the positioning system 108 may be configured to merge one or more spatial anchors observed through the AR glasses (i.e., the coordinates of one or more spatial anchors in the augmented reality coordinate system associated with the AR glasses) with the real-world spatial coordinates of the real-world objects to which the spatial anchors belong, which are stored in the database system in the form of a spatial mapping grid, to obtain the real-world spatial position of the user of the AR glasses in the factory.

[0177] In another embodiment, the positioning system 108 is an indoor positioning system. The AR glasses 104 and / or the program logic 106 are configured to receive signals from Bluetooth beacons distributed throughout the factory, and determine the location based on the received signal strength indication (RSSI) of the received Bluetooth beacon signal. Alternatively, and particularly advantageous in a factory environment with a large number of obstacles (i.e., non-open floor layouts) that may cause multipath effects, shadow fading, and signal blocking, the AR glasses 104 and / or the program logic 106 are configured to receive cellular network and Bluetooth beacon signals, as described in "Indoor Positioning System With Cellular Network Assistance Based on Received Signal Strength Indication of Beacon" on pages 6691-6703 of IEEE Access, December 30, 2019, which is incorporated herein by reference. In this embodiment, the RSSI value of the received Bluetooth beacon signal is signal processed (via filtering to replace missing values ​​and remove outliers) to generate a smoother signal within the range. Then, the cell where the AR glasses are located is determined based on the four strongest RSSI values, and a weighted multi-point positioning algorithm with cellular network assistance is used to calculate the real-time position of the AR glasses (i.e., the position of the AR user).

[0178] In yet another embodiment, the positioning system 108 is a cellular positioning system based on the 5G mobile radio standard. The 5G positioning system provides extremely high network reliability and high positioning accuracy. In another embodiment, the positioning system is a GPS positioning system, wherein the AR glasses include a GPS receiver and corresponding software and / or GPS applications configured to determine the spatial position of the AR glasses.

[0179] The executable program logic 106 is further configured to receive data corresponding to equipment located near the latest spatial location of the user from the database system 102, and control the augmented reality glasses 104 to display at least some of the received data. In one embodiment, the executable program logic 106 is configured to receive data corresponding to any object of the plant located near the latest spatial location of the user from the database system 102.

[0180] In one embodiment, when the latest spatial position of the user wearing the AR glasses 104 is received from the positioning system 108, the executable program logic 106 determines the coordinates of the bounding space near the user wearing the AR glasses 104. According to the embodiment, the executable program logic 106 is configured to determine the coordinates of the bounding space near the user based on one or more of the following bounding space parameters: the latest spatial position of the user, the user's line of sight, the number of equipment near the user or in the user's line of sight (e.g., the density of equipment near the user and / or in the user's line of sight), the number and / or data type of virtual objects corresponding to the equipment near the user and / or in the user's line of sight, the configuration of the walls, floors and / or ceilings of the factory infrastructure near the user, and one or more user roles assigned to the user wearing the AR glasses 104. The bounding space is thus defined by the coordinates of the bounding surface defining the bounding space, as determined by the program logic. The program logic can also be configured to interpolate between the determined coordinates to add more coordinates that define the bounding surface of the bounding space at a higher resolution.

[0181] According to an embodiment, a boundary space is a 3D boundary space located within (i.e., defined by) a boundary surface or a 2D boundary space located within (i.e., defined by a boundary "surface" in a plane. Therefore, for the purpose of describing embodiments of the present disclosure, the term "boundary surface" is defined as a 2D surface defined by a set of coordinates that encloses a 3D boundary space or a 1D "surface" defined by a set of coordinates that encloses a 2D boundary space (i.e., as an illustrative non-limiting example, a continuous line, such as a curve that defines the circumference of a circle). In addition, for the purpose of describing embodiments of the present disclosure, the term "boundary surface" is defined to include a 3D boundary space and a 2D boundary space.

[0182] In one embodiment, if the user's most recent spatial position places the user outside the factory boundary or within an area designated as off-limits for reasons such as protecting proprietary information or limiting access to extremely hazardous areas (e.g., high voltage, air pollutants, infrastructure repairs), the program logic determines that the boundary space does not exist, or alternatively, the boundary space has a very limited length in a particular direction, such as, for example, a direction pointing to a hazardous area or a highly appropriate area. Thus, the program logic can be configured to determine the coordinates of the boundary space based on the most recent spatial position of the user wearing the AR glasses 104.

[0183] In another embodiment, the program logic may determine and use the line of sight between the user wearing the AR glasses and the virtual object to determine whether the virtual object should be displayed. For example, the executable program logic may dynamically receive position information of one or more physical objects such as equipment, materials, and / or products. The executable program logic may determine that one or more of these physical objects are located in the line of sight between the user and the virtual object to be displayed. In one embodiment, if the program logic determines that one or more physical objects in the user's line of sight completely or partially blocks the user's line of sight, the control logic does not display the hidden portion of the virtual object. If the virtual object is completely hidden by the physical object in the line of sight, the virtual object is not displayed. For example, the ratio of the hidden portion of the virtual object can be defined.

[0184] In another embodiment, the program logic can determine the coordinates of the boundary space based on the density of equipment near the user. For example, the density of equipment near the user (e.g., equipment surrounding the user and close to the user) and / or the density of equipment in the user's viewing direction (e.g., equipment close to the user and in the user's viewing direction determined by the AR glasses orientation or eye orientation) can be defined as the number of equipment per unit volume of the three-dimensional boundary space and the number of equipment per unit area of ​​the two-dimensional boundary space. The equipment density value can be associated with the spatial position in the factory and stored in, for example, the database 130 of the database system 102, which can be retrieved by the program logic 106. In one embodiment, the boundary radius parameter is determined based on the density of the equipment. For example, the boundary radius parameter can be inversely proportional to the density of the equipment. Therefore, when the spatial position of the user wearing AR glasses corresponds to a higher than average density of the equipment, the program logic provides a lower than average boundary radius parameter. Once the boundary radius is determined, the program logic uses, for example, the boundary radius parameter to define an initial spherical boundary surface (or an initial hemispherical boundary surface, where the flat surface of the hemisphere coincides with the floor of the factory). The program logic may then modify the initial spherical boundary surface (or initial circular boundary surface for embodiments with a 2D boundary space) having an initial radius equal to the boundary radius parameter based on other boundary space parameters, such as the user's most recent spatial position and / or the orientation of the AR glasses and / or the orientation of the user's eyes, as discussed above. That is, the program logic may modify the initial spherical boundary surface to form a 3D non-spherical boundary surface whose coordinates are further based on any of the other boundary space parameters, as discussed above and further below. By making the boundary radius parameter inversely proportional to the density of equipment, embodiments of the present disclosure limit the amount of information provided to the user of the AR glasses at the same time, so as not to overwhelm the user with too much data and to preserve system bandwidth for other tasks.

[0185] In another embodiment, the program logic may determine the coordinates of the bounding space based on the number and / or data type of virtual objects corresponding to equipment near the user and / or in the user's viewing direction. For example, the program logic may determine the total number of virtual objects corresponding to equipment near the user. In one embodiment, the program logic determines the total number of virtual objects based on the radius of a sphere centered on the user, in which the corresponding equipment is located. Then, based on a predefined virtual object maximum parameter that can be set and / or adjusted by the operator, the program logic determines the maximum radius so that the difference between the total number of virtual objects and the virtual object maximum is minimized, and the total number of virtual objects does not exceed the virtual object maximum.

[0186] In another embodiment, given a user spatial position (e.g., the latest spatial position), the program logic determines the total number of virtual objects based on the distance from the user measured along the viewing direction of the user wearing the AR glasses, and the corresponding equipment corresponding to the total number of virtual objects is located within the distance. Then, based on a predefined line of sight virtual object maximum parameter that can be set and / or adjusted by the operator, the program logic determines the maximum distance so that the difference between the total number of virtual objects and the line of sight virtual object maximum value is minimized, and the total number of virtual objects does not exceed the line of sight virtual object maximum value. In another embodiment, the program logic determines multiple maximum distances, one maximum distance for each different viewing direction of the user at a given user spatial position, wherein each maximum distance is determined based on the use of the line of sight virtual object maximum value parameter discussed above. The multiple maximum distances along the corresponding multiple lines of sight then define multiple coordinates on the boundary surface, or in other words, define the boundary surface and therefore define the boundary space (i.e., the space surrounded by the boundary surface). The boundary surface can have any shape and depends only on the multiple maximum distances determined along the multiple lines of sight.

[0187] The data type of the virtual object may be, for example, one of video data, text data, and audio data. Another data type is alarm data. Alarm data is data that the user wearing the AR glasses urgently needs to know. Therefore, the alarm data may be represented by audio data (e.g., beeps or sirens), text data (e.g., flashing red text), or video data (e.g., short looping video messages), or any combination of two or more of the audio data, text data, and video data. In one embodiment, the display of a virtual object consisting of alarm data by AR glasses does not require action on the part of the user. Actions include, but are not limited to, a user "clicking" a virtual object text box, which indicates that text data, video data, or audio data corresponding to another virtual object is available for display. In order to distinguish audio data, text data, and video data used as alarm data from audio data, text data, and video data that are not used as alarm data, the audio data, text data, and video data used as alarm data are also referred to as emergency audio data, emergency text data, and emergency video data for the purposes of this disclosure.

[0188] In one embodiment, program logic determines the coordinates of the boundary space based on the data type of the virtual object of the equipment corresponding to the user's vicinity. For example, compared with the virtual object composed of video data, text data or audio data, the virtual object composed of alarm data can be (by the operator of the system) pre-assigned with a higher display priority. The virtual composed of video text or audio data can also be pre-assigned with a display priority value according to the type of data (for example, in one embodiment, audio data has a higher display priority than text data, and text data has a higher display priority than video data). In one embodiment, the total display priority value at any given sight distance from the user is defined as the sum of the display priority values ​​of the corresponding virtual objects associated with the equipment placed along the sight line between the user and the given sight line distance. The operator can also define the maximum display priority value. In one embodiment, program logic determines the maximum distance along the user's sight line so that the difference between the total display priority value and the maximum display priority value is minimized, and the total display priority value does not exceed the maximum display priority value. In another embodiment, program logic determines multiple maximum distances based on the display priority value, and a maximum distance is for each different sight line of the user at a given user space position. A plurality of maximum distances based on the display priority values ​​along the corresponding plurality of sight lines define a plurality of coordinates on the boundary surface, or in other words, define the boundary surface.

[0189] In another embodiment, the boundary space is one of two or more data type specific boundary spaces. As discussed above, the virtual object includes an object of a data type selected from a group consisting of a video, an image, a text, an audio file, and an alarm message. The executable program logic 106 is configured to dynamically determine the coordinates of the data type specific boundary space based on the data type of the virtual object to be displayed to the user wearing the AR glasses. For example, the user can select the data type specific boundary space to be used by the system via an audio command to the AR glasses, via a selection of a command displayed to the user by the AR glasses, for example, during the initial setting of the AR glasses, or via the selection of the operator of the system. That is, if the specific boundary space is selected (or identified) as an alarm message specific boundary space (also referred to as an alarm boundary space), the program logic determines the coordinates of the boundary space based on the constraint that a virtual object consisting only of an alarm message will be displayed to the wearer of the AR glasses (ie, the user). Therefore, the program logic is configured to determine the maximum distance of the line of sight of multiple lines of sight based on the user's latest spatial position, for defining the radius of the boundary space and / or the boundary space surrounding and close to the user based only on the boundary space including a virtual object consisting of an alarm message, as described above.

[0190] Advantageously, the program logic provides an alarm message specific boundary space, the size and / or shape of which can contain a larger volume (or area) for displaying a greater number of alarm messages than would be displayed if the boundary space was not identified as a data specific boundary space. In other words, the program logic provides a data specific boundary space having a size and / or shape different from the size and / or shape of a non-data specific boundary space (i.e., a boundary space of virtual objects including all data types). In this way, the size and / or shape of the boundary space is customized to be data specific, thereby providing a boundary space that is "optimized" for viewing virtual objects of a specific data type or for viewing virtual objects composed of any combination of one or more data types in other embodiments. In the sense of the present disclosure and in accordance with the described embodiments, "optimization" means that the size and / or shape of the boundary space is determined to provide the user with the most meaningful and / or relevant data that the user should be aware of and can be assimilated by the user, without exceeding the user's cognitive ability to process information visually and / or auditorily. The "size and / or shape" of the term boundary space and the "coordinates" of the boundary space used to describe the embodiments disclosed throughout the present application are defined to have the same meaning. That is, once the coordinates of the bounding space are determined to be the same as the coordinates of the bounding surface surrounding the bounding space, the size and / or shape of the bounding space are also determined.

[0191] In another embodiment, the program logic determines the coordinates of the boundary space based on the configuration of the wall, floor and / or ceiling of the factory infrastructure near the user. For example, the program logic is configured to determine the coordinates corresponding to the height of the boundary space based on the height of the room or area relative to the spatial position of the user. If the user is in a closed or semi-enclosed space (such as a room), the program logic may also be configured to determine one or more lateral dimensions (i.e., lateral to the dimension of height) of the boundary space. In one embodiment, the program logic determines the height of the room, compartment or area close to and / or completely or partially surrounding the user by receiving measurement data from one or more sensors worn by the user. Alternatively or in addition, the program logic determines the height of the room, compartment or area close to and / or completely or partially surrounding the user according to the room ID, compartment ID or area ID detected by the sensor worn by the user. For example, the room, compartment and / or area (such as wall and / or ceiling) of the factory and the equipment in some embodiments may have an electronically readable (e.g., scannable) ID tag. The AR glasses may include a scanner, such as a scanner with QR code capabilities, and the program logic may store an allocation table that assigns the height to the room ID and / or compartment ID and / or area ID.

[0192] In another embodiment, the program logic determines the coordinates of the boundary space based on the configuration of the walls, floors, and / or ceilings of the plant infrastructure near the user in combination with one or more user roles assigned to the user. For example, the executable program logic is configured to receive user credentials from a user wearing glasses and authenticate the user associated with the user ID. The user may be a new user who needs to be authenticated, or the user may already have a registered user account in a user database, including, for example, a user ID. The user ID may be associated with one or more user roles, such as an "operator" role or a "maintenance" role. Additionally or alternatively, each user role may optionally include one or more user permissions, as discussed previously.

[0193] According to one embodiment, the program logic is configured to receive one or more user roles associated with a user ID. For example, the user role can be the role of an "operator", that is, a user who is obliged to control and / or monitor one or more steps in a production workflow, or the role of a "maintenance worker", that is, a user who is obliged to perform maintenance words to ensure that at least one equipment is working properly. The program logic is configured to determine the scope of the boundary space based on the role (or multiple roles) assigned to the user ID, such as determining whether the boundary space includes more than one room or more than one floor of the factory, and / or the scope of the boundary space close to the user at any spatial location in the factory. For example, if a user wearing AR glasses has a maintenance role instead of an operator role, virtual objects associated with equipment in multiple different rooms and / or floors of an industrial plant relative to the user's latest spatial position can be presented to the user. Generally speaking, because maintenance workers are responsible for technically complex tasks that may require in-depth technical knowledge of equipment that is typically unavailable or even unnecessary for users with the "Operator" role, users with the Maintenance role can have a bounding space of much larger size and / or shape to include virtual objects associated with a wider variety of equipment, even equipment that is not located close to the user's most recent spatial location, such as equipment located in different rooms or on different floors of a factory.

[0194] The scope of the present disclosure also includes implementations in which program logic uses one or more of the above-mentioned boundary space parameters in any combination when determining coordinates of a boundary space near a user wearing AR glasses (i.e., coordinates defining a boundary surface that encloses the boundary space), such as the user's most recent spatial position, the orientation of the user's AR glasses, the orientation of the user's eyes, the user's line of sight, the number of equipment near the user or in the user's line of sight (e.g., the density of equipment near the user and / or in the user's line of sight), the number and / or data type of virtual objects corresponding to equipment near the user and / or in the user's line of sight, the configuration of walls, floors and / or ceilings of factory infrastructure near the user, and one or more user roles assigned to the user wearing the AR glasses 104.

[0195] In another embodiment, the program logic determines whether the boundary space determined by any of the embodiments described above includes one or more equipment that is not completely within the boundary space. For example, once the program logic determines the boundary space defined by the boundary surface surrounding the boundary space, the program logic determines whether any equipment with at least one corresponding virtual object located within the boundary space has at least one coordinate located outside the boundary space. Equipment defined by more than one coordinate (for example, by the coordinates defining the corners of the equipment) can be bisected by the boundary surface so that some of the coordinates corresponding to the equipment are located within the boundary space, while other coordinates corresponding to the equipment are located outside the boundary space. In one embodiment, the program logic adjusts the boundary surface that bisects any equipment so that the equipment is completely included in the boundary space, and therefore also adjusts the display to include those virtual objects included in the newly adjusted boundary space. In another embodiment, the program logic adjusts the boundary surface that bisects any equipment so that the equipment is completely outside the boundary space, and therefore also adjusts the display to include only those virtual objects included in the newly adjusted boundary space.

[0196] Once the program logic 106 has determined the coordinates of the bounding space that is proximate to the user wearing the AR glasses 104, or has coordinates of at least a portion of the bounding space that is not proximate to the user wearing the AR glasses 104 (which may occur when the program logic uses one or more user characters to determine the coordinates of the bounding space), the program logic 106 displays one or more virtual objects to the wearer of the glasses via the AR glasses 104 if and only if the coordinates of the virtual objects are within the bounding space. The displayed virtual objects include data, such as text data, video data, image data, audio data, or alarm data. The data is associated with monitoring and / or controlling an automated production process.

[0197] For example, in one embodiment, the program logic sends the latest spatial position of the user wearing the AR glasses and the coordinates of the bounding space (e.g., the coordinates of the bounding surface that encloses the bounding space) to the database system 102, and receives from the database system those virtual objects having coordinates contained within the bounding space defined by the coordinates of the bounding space. Thus, the user of the AR glasses (e.g., the wearer) views the overlay of the virtual objects contained within the bounding space with the real-world objects of the factory. The user can then use the text data, image data, video data, audio data, and / or alarm data corresponding to the displayed virtual objects to maintain and / or control the corresponding equipment to which the virtual objects correspond, and / or obtain operating status, parameter settings, and / or educate themselves on how the equipment operates and / or the roles of different equipment in the production process.

[0198] In addition, the program logic is configured to continuously update the boundary space and the coordinates of the virtual objects displayed therein according to the latest spatial position of the user wearing the AR glasses. In addition, the coordinates of the boundary space are determined according to the orientation of the AR glasses and / or the orientation of the eyes (gaze) of the user wearing the AR glasses. Therefore, when the user wearing or operating the AR glasses moves to a different spatial position, and / or when the user (although stationary relative to the spatial position) changes the line of sight through the glasses by changing the orientation of the AR glasses (for example, the user wearing the AR glasses as a head-mounted component moves the head-mounted component by moving the user's head), the program logic determines a new boundary space. Then, if and only if the coordinates of the virtual object are within the new boundary space, the program logic 106 displays one or more virtual objects to the wearer of the glasses via the AR glasses 104. The virtual objects displayed in the new (i.e., updated) boundary space may include some of the virtual objects displayed in the previous boundary space, depending on any overlap between the new boundary space and the previous boundary space.

[0199] In another embodiment, the program logic is configured to continuously update the boundary space and the coordinates of the virtual objects displayed therein according to the latest spatial position of the user wearing the AR glasses. According to some embodiments, data indicating the orientation of the AR glasses is also used to determine and update the boundary space. For example, the AR glasses may include a gyroscope and / or an accelerometer for determining the orientation of the AR glasses. In addition or alternatively, the executable program logic determines and updates the coordinates of the boundary space according to the orientation of the eyes of the user wearing the AR glasses. A variety of methods and corresponding sensors for determining the orientation ("gaze") of the user's eyes are known and can be used in embodiments of the present invention. For example, for an overview of existing eye tracking systems, see Al-Rahayfeh A, Faezipour M. "Eye Tracking and Head Movement Detection: A State-of-Art Survey", IEEE J Transl Eng Health Med. 2013; 1: 2100212. It was published on November 6, 2013, doi: 10.1109 / JTEHM.2013.2289879.

[0200] Figures 1B to 1HA top view of AR glasses 104 worn on a user's head 105 is shown. The user looks through the AR glasses with the user's eyes 107. The orientation of the eye (i.e., the direction the user is looking, or in other words, the direction the user is gazing) is defined by an eye orientation direction 109 that is perpendicular to the pupil 111 of the eye. The AR glasses have an orientation direction 113 defined by the orientation of the user's head 105. This application uses the terms "user's line of sight" and "viewing direction" interchangeably. The viewing direction can be determined, for example, by detecting the "orientation" of the AR glasses and / or by detecting the orientation of the eyes, for example, with the aid of eye tracking technology.

[0201] For example, in an embodiment where the AR glasses do not include sensors for detecting the orientation of the user's eyes, the orientation of the AR glasses can be used as an indicator and / or predictor of the user's viewing direction. According to other embodiments, the AR glasses or other components of the automated system external to the AR glasses may include one or more sensors, such as cameras, for detecting the orientation of the user's eyes. In this case, the orientation of the user's eyes detected by these sensors is transmitted to the executable program logic and used to determine the coordinates of the boundary space.

[0202] Figure 1B The orientation 113 of the AR glasses is shown pointing in the same direction as the eye orientation 109 of the user wearing the AR glasses. Figure 1C and Figure 1D It is shown that the orientation 113 of the AR glasses and the eye orientation 109 of the user wearing the AR glasses point in different directions. For example, Figure 1C The orientation 113 of the AR glasses is shown pointing straight ahead when the user is gazing to the right (i.e., gazing through the right portion of the glasses), and Figure 1D The orientation 113 of the AR glasses is shown pointing straight ahead when the user is gazing to the left (ie, gazing through the left portion of the glasses).

[0203] In one embodiment, the positioning system 108 includes an optical sensor system that includes a camera or other optical device, such as a laser, that measures the position of the pupil in the eye to determine the eye orientation (also referred to as the eye orientation direction) of the user wearing the AR glasses. As discussed above, the orientation of the AR glasses worn by the user is not necessarily the same as the user's eye orientation, and in some embodiments, the coordinates of the bounding space depend not only on the spatial position of the user wearing the AR glasses, but also on the orientation of the AR glasses (i.e., the orientation of the user's head) and / or the user's eye orientation (i.e., the direction in which the user is looking through the glasses).

[0204] Figure 1EThe boundary space around the user wearing AR glasses 104 according to one embodiment is shown. The boundary space 150 can be a 3D space, and the cross section of the boundary space surface is shown as a dotted line. According to the embodiment, the boundary shape can have different forms, such as a sphere, a hemisphere, a polygon-based 3D body, a cube, a cuboid, etc. The executable program logic 106 is configured to display the virtual object to the user via the AR glasses 104 only when the virtual object is located within the boundary space 150. For example, the virtual object 115 located within the boundary space 150 is displayed, and the virtual object 117 is not displayed via the AR glasses. Object 117 is displayed only when the user approaches the position of the virtual object 117 of the augmented reality coordinate system (or the object approaches the user) so that the virtual object 117 is within the boundary space 116. The boundary space can be dynamically updated according to the current position of the user wearing the glasses, so that the user is always at a predefined position within the boundary space, such as at the center, at a point within the portion of the boundary space facing the rear side of the user, or at another position.

[0205] Figure 1F FIG. 1 shows a boundary space that is close to but not surrounding a user wearing AR glasses according to another embodiment. In the depicted embodiment, the boundary space has a cubic form. In other aspects, the boundary space may have, for example, a Figure 1E The properties described in the embodiment depicted in .

[0206] Figure 1G Two different bounding spaces for two different types of virtual objects are shown. For example, bounding space 150 may have a smaller volume than bounding space 152. Bounding space 150 may be completely included in bounding space 152 or may overlap with the bounding space, and may have the same or different forms. In the depicted example, bounding space 152 is a body with a trapezoid as its base, while bounding space 150 is an ellipsoid. In the depicted example, the executable program logic uses bounding space 150 to determine whether to display a virtual object of the "video" type via the AR glasses: if a virtual object of the "video" type is located within bounding space 150, the virtual object is displayed as in the case of object 115. Otherwise, the virtual object is not displayed as in the case of object 117. The executable program logic uses bounding space 152 to determine whether to display a virtual object of the "alarm message" type via the AR glasses (or issue via the acoustic output interface in the case of an acoustic alarm): Alarm messages 118 are displayed and / or output via the AR glasses because they are located within the alarm type bounding space 152. The alert 119 is not displayed / output because it is outside the boundary space 152 .

[0207] This may be advantageous because human users cannot usually watch multiple videos in parallel. For example, by only displaying instructional videos about how to operate a specific machine in close proximity to the user, the user is prevented from being distracted and making mistakes when operating the machine. It also reduces the amount of data that needs to be transmitted to the AR glasses and avoids the need for computing power to render and display multiple videos. On the contrary, displaying alarm messages in a much larger space has the following advantages: even if the user is not in the immediate vicinity of the machine, important status messages that are necessary for the manufacturing process can be transmitted to the user. In some embodiments, the boundary space of the alarm message is so large that the user can even be in another room or on another floor. Alarm messages can usually be processed quickly by the user and only cause a small burden on the bandwidth and computing power of the AR system. Therefore, using different boundary spaces for different types of virtual objects (e.g., videos, text, alarm messages, etc.) can significantly improve the safety and efficiency of monitoring the maintenance of the manufacturing process or related machines.

[0208] According to an embodiment, the size of a virtual object-specific boundary space is selected so as to ensure that virtual objects displayed within the boundary space can be perceived and interpreted by the human visual system: at the edge of the field of view, the eye has only lower vision. Therefore, displaying complex, finely constructed virtual objects (e.g., objects with a lot of text) at the edge of the field of view has little benefit to the user and may even distract the user and consume CPU resources. By generating and using a boundary space adapted to the type of virtual object by executable program logic to decide whether a given virtual object should be displayed, operational errors are avoided.

[0209] Figure 1H A bounding space is shown whose position and / or orientation dynamically adapts to the gaze of a user wearing the AR glasses. In the depicted embodiment, the AR glasses include a sensor for detecting the user's gaze independently of the orientation of the AR glasses. In this case, the position and / or orientation of the bounding space (or multiple bounding spaces, if applicable) can be dynamically updated so that it follows and reflects changes in the user's gaze direction. For example, the update may include repositioning the bounding space so that its center is always at a point on the user's line of sight (here: gaze). Thus, before the user moves their gaze to the right, as in Figure 1G As shown, virtual object 115 is shown, but 117 is not shown. After the user has moved his gaze to the right, as shown in FIG. Figure 1H As shown, virtual object 115 is shown, but 117 is not shown.

[0210] According to other embodiments, the position and / or orientation of a bounding space (or multiple bounding spaces, if applicable) is dynamically updated so that it follows and reflects changes in the orientation of the AR glasses worn by the user and / or changes in the user's gaze (orientation of the eyes).

[0211] According to yet another embodiment, the database system 102 is a graph database system, a spatial database system and / or a streaming database system. The database system 102 further includes a database 110 .

[0212] In one embodiment, the database system 102 is a spatial database system, and the database 110 is a spatial database configured to store the corresponding data based on the spatial coordinates associated with the corresponding data, and wherein the corresponding data is searched and retrieved based on the spatial coordinates associated with the corresponding data. For example, the spatial database system 102 of the present invention stores a plurality of spatial coordinates, wherein each spatial coordinate includes corresponding data related to an object located at the spatial coordinate in the factory. For example, given a specific equipment (e.g., a pump) of a factory, the spatial database stores one or more spatial coordinates of the pump, an optional description of each spatial coordinate (e.g., spatial coordinates 1-4 represent the four corners of the pump on the floor of the factory, spatial coordinate 5 represents the high point of the pump, etc.), and data associated with the pump (e.g., task instructions for the pump). By way of an exemplary embodiment, the task instructions (of the pump) may include, but are not limited to: one or more training videos in the training video, one or more photos and / or training photos, related assets (e.g., other equipment or factory objects that can accommodate pumps or operate in conjunction with pumps, such as equipment that provides raw materials to the pump or accepts the outflow of products or pumps), short task instruction text, detailed task instruction text, and a list of task instruction categories. The following will be combined with Figure 2 Task instructions are discussed further. In particular, database system 102 may be a graph database system that supports fast geometric operations, ie, a spatial graph database, which supports fast in-depth analysis of equipment-related data, including fast determination of distances to other objects and / or users.

[0213] In one embodiment, the executable program logic 106 is configured to perform a query to the spatial database 110 based on the latest spatial location to retrieve data from the spatial database 110 for equipment near the user's latest spatial location. The query can be, for example, a conventional SQL query that is repeated multiple times using the user's current location as an independent variable, or can be a subscription to a streaming service that is updated once the user changes their location. In one embodiment, the proximity of the location is defined by a user-adjustable proximity parameter of the executable program logic 106. The user can set the proximity parameter to any radius measured from the user's spatial location or to any radius defining a hemisphere, where the flat surface of the hemisphere is positioned on the floor of the factory and centered on the user's latest spatial location.

[0214] In another embodiment, the executable program logic 106 is configured to perform a query to the spatial database 110 based on the most recent spatial position used and based on the user's field of view through the AR glasses to retrieve data from the spatial database 110 for equipment that is near the most recent spatial position of the user of the AR glasses and within the field of view of the user of the AR glasses. For example, the proximity of the location is defined by a user-adjustable proximity parameter of the executable program logic 106. The user can set the proximity parameter to any radius measured from the user's most recent spatial position or to any radius defining a hemisphere, where the flat surface of the hemisphere is positioned on the floor of the factory and centered on the user's spatial position. In response to the query, the executable program logic 106 receives data for equipment that is located within a 3D cone defined by the current field of view of the AR glasses and having a length defined by the user-defined radius.

[0215] In one embodiment, data of equipment located near the user's most recent spatial location and optionally within the field of view of the user of the AR glasses is received by the executable program logic, even if such data is blocked from view by obstacles in the plant and is not visible to the user of the AR glasses. Therefore, if the second equipment is partially or completely hidden behind the first equipment, the executable program logic 106 is configured to control the augmented reality glasses 104 to display the data associated with the second equipment (i.e., the second object) by placing the data associated with the second equipment near the second equipment so that it is not obscured and cannot be viewed by the first equipment. In one embodiment, the data associated with the second equipment is displayed with a smaller font and / or a smaller image and / or a pointer pointing in the direction of the hidden or partially hidden second equipment to indicate that the data is associated with equipment that is at least partially hidden outside the field of view of the user of the AR glasses 104.

[0216] In another embodiment, the spatial database 110 of the spatial database system 102 is accessible via a subscription-based streaming service. The executable program logic 106 is configured to receive data of equipment located near the latest spatial position of the user of the AR glasses from the spatial database 110 via the streaming service. That is, in contrast to retrieving data of equipment located near the latest spatial position of the user from the spatial database 110 via a query of the spatial database 110 as discussed above, according to this embodiment, the executable program logic 106 and the spatial database 110 are configured so that the executable program logic 106 receives data of equipment located near the latest spatial position of the user via the service of any commercially available third-party streaming service based on the latest spatial position of the user transmitted to the streaming service provider. The spatial database system 102 including the spatial database 110 that can be accessed via any subscription-based streaming service may also be referred to as a streaming spatial database system.

[0217] According to some embodiments, the database system 102 is a graph database system, and the database 110 is a graph database that supports fast geometric operations. The database system can therefore be considered a combined graph database and a spatial database. The graph database is configured so that nodes represent objects in the physical world, in particular equipment and equipment parts or sub-parts, and edges represent the real-world space and organizational relationships of these equipment, parts and sub-parts. Data associated with equipment and parts, such as a manual or GUI for setting process parameters or location information of the equipment or parts, can be stored, searched and retrieved based on spatial coordinates associated with the corresponding data, and the executable program logic 106 is configured to perform a query of the graph database based on the latest spatial position and optional field of view of the AR glasses 104, for retrieving data from the graph database of equipment located near the user's latest spatial position and optionally within the field of view (e.g., within the field of view 3D cone), the field of view being defined by a reference 3D cone, the vertex of which is located at the AR glasses, and the longitudinal axis of symmetry points to the viewing direction of the user of the AR glasses.

[0218] According to some embodiments, the graph database 110 of the graph database system 102 can be accessed via a subscription-based streaming service, as described above in conjunction with the spatial database. In this embodiment, the executable program logic 106 is configured to receive data of equipment located near the latest spatial position of the user of the AR glasses 104 from the graph database 110 via the streaming service. The graph database system 102 including the graph database 110 that can be accessed via any subscription-based streaming service can also be referred to as a streaming graph database system.

[0219] The graph database (and streaming graph database) can be configured so that the objects of the factory are stored as interconnected nodes of the graph, where each node includes one or more spatial coordinates of an object in the factory and data corresponding to the object. Given the architecture of the graph database, the graph database (and streaming graph database) of the present invention enables rapid search and access to data associated with a given object in the factory along the topology of the graph, thereby evaluating the spatial coordinates of the user of the AR glasses in the factory.

[0220] In one embodiment, objects in a factory include, but are not limited to: one or more sensors, where sensors 112, 114, and 116 are exemplary embodiments; one or more actuators, where actuators 118 and 120 are exemplary embodiments; at least one equipment 122, 124, such as a machine (e.g., a robot or a stationary manufacturing machine); a machine component or subcomponent; a tool or; and / or raw materials 126 and 128. In some embodiments, one or more actuators and / or sensors (such as actuators 118, 120 and sensors 114, 116) are connected to factory equipment 122 124 for actuating the equipment and / or components of the equipment to sense parameters associated with the equipment and / or raw materials 126, 128 processed by the equipment. Parameters that may be sensed include, but are not limited to, pressure, temperature, humidity, and parameters associated with the movement of equipment (e.g., a machine such as a stationary machine or a robot) or components of the equipment, or raw materials being processed or to be processed by the equipment (e.g., conveyor speed, fluid flow rate in a pipe, rotation speed of a mixer, movement of an automation component, etc.). The scope of the present invention also includes sensors, such as sensor 112, that are not connected to any factory equipment, for measuring ambient factory conditions, such as, for example, temperature within the factory, ambient lighting, or particle concentration in the air. The scope of the present invention also includes objects in the factory that are not connected to any sensor or actuator, such as equipment 122 and robots 124. Factory equipment may also include inactive (i.e., non-participating) structural components of the factory (i.e., factory components such as, for example, racks, walls, ceilings, heaters, air conditioners, and support structures that are not involved in the processing of raw or intermediate materials).

[0221] In another embodiment, the database system 102 optionally includes an isolated virtual user space, a so-called "container" 130. For example, the container can be implemented as a Docker container. Each container 130 is associated with a corresponding node and is configured to continuously determine the current spatial information associated with the object of the corresponding node (e.g., the position and / or orientation of the object) and immediately update the position information of the object in the database 110. For example, each container may include a software function that can interoperate with the positioning system 108 and is configured to continuously receive the current position information of the corresponding object. In addition or alternatively, one or more containers in the container may include a software function that can interoperate with one or more position determination sensors attached to the object represented by the node associated with the container, and can directly receive position information from the position determination sensor.

[0222] According to a preferred embodiment, the database 110 or another data repository includes a configuration of the factory, which specifies the types and locations of multiple physical objects that make up the factory, such as equipment, equipment parts, and optionally raw materials and / or products. The container 130, in combination with the position sensor and / or position determination system 108, advantageously enables the configuration of the factory to be updated fully automatically in response to any movement and / or rearrangement of objects in the factory. For example, robots can move around the factory, and components of the automated system can also be moved to different locations as part of their operation. Because the containers are isolated from each other, each container is a "standardized software unit" and includes all of its libraries and dependencies required for all software programs running in the container, including automated position determination and position information update functions associated with the corresponding objects of the factory, so that the software within the container can be reliably executed regardless of the architecture of the database system and regardless of the state of other containers and corresponding objects.

[0223] According to some embodiments, for example Figure 1A In the embodiment depicted in , at least one device includes one or more robots. One or more of the robots can be configured to perform one or more tasks in the manufacturing process, such as delivering raw materials to a machine or cutting or gluing materials. Additionally or alternatively, one or more of the robots can be configured to move and act as a physical representative of a remote user, such as Figure 7 The robot may include an interface for receiving control commands from a robot control module external to the robot, whereby the control commands determine the position, orientation, movement, or task performed by the robot. According to some examples, the control commands are suitable for controlling the movement, position, and / or orientation of the robot such that the movement, position, and / or orientation of the robot corresponds to and reflects the movement, position, and / or orientation of the avatar of the remote user within a coordinate system associated with the augmented reality glasses.

[0224] According to some examples, the robot representing the remote user and acting as a physical representation of the remote user includes one or more sensors (such as a camera and a microphone) configured to capture sensor data (e.g., images, video, captured audio signals) and submit the sensor data to the virtual reality application 701 (see Figure 7), so that the virtual reality application can generate virtual reality based on the sensor data, so that the remote user sees the images captured by the robot's camera and hears the acoustic signals captured by the robot's microphone in real time. According to a preferred embodiment, the virtual reality application 701 and the robot control module are configured to interoperate, thereby achieving two-way real-time synchronization between the virtual reality presented to the remote user and the robot's physical environment: the movement of the remote user (or at least the remote user's head triggers the corresponding movement of the robot, and the optical signals and acoustic signals acquired by the robot's sensors are provided to the VR application and used by the VR application to create and display and / or acoustically output a virtual world to the remote user, whereby at least a portion of the virtual world reflects the acquired acoustic and optical signals.

[0225] According to some examples, the robot includes a first interface for receiving control commands from a robot control module of the control system 132 and for sending sensor data to the control system 132. This can have the following advantages: the control system 132, which can be configured to coordinate multiple equipment during a manufacturing process, can use and control the robot to perform specific tasks of the manufacturing process.

[0226] According to some examples, the robot includes a second interface (e.g., such as a control interface) for receiving control commands from the AR system 101 (e.g., from the AR application 106 and / or from the VR application 701 included in the AR system). Figure 7 As shown). For example, the AR system 101 may also include a robot control module configured to receive sensor data from the robot and / or send control commands to the robot for controlling the movement, orientation, position of the robot and / or the tasks performed by the robot. The current position and orientation of the robot are continuously reported and stored in the database 110. If the robot changes position and the updated coordinates of the robot are stored in the database, the event may trigger the display or hiding of a virtual object in the AR glasses of the local user. This may have the following advantages: the robot is seamlessly integrated into the manufacturing process coordinated by the control system via the first interface, and is also integrated into the world of virtual objects created by the AR application or VR application. In some examples, the robot may include a conflict resolution module for resolving conflicts that may be caused by incompatible control commands received via the first interface and the second interface. For example, the conflict resolution module may resolve conflicts by overwriting or ignoring conflicting commands provided by the remote user's VR application instead of executing conflicting control commands received via the first interface.

[0227] The database system 102 is also considered to store location information and other data describing physical objects of the factory, such that the data in the database system represents a digital twin, in the sense that the database system 102 is configured to compute or allow a complete copy of the factory and its components to be computed by the executable program logic 106. The virtual copy of the factory may include all factory objects, or alternatively include those factory objects that have an impact on the production process, and may include one or more real-world space coordinates associated with each factory object, as well as data corresponding to each factory object.

[0228] According to another embodiment, the spatial information associated with the object of the corresponding node includes one or more spatial coordinates of the object. For example, in other embodiments, one or more spatial coordinates correspond to one or more points on the boundary surface of the object and / or in the boundary surface of the object. The boundary surface of the object is defined as the outer surface of the packaging object including the object. The spatial coordinates in the boundary surface can be, for example, the centroid of the object or the geometric center of the object. The packaging object can be a rectangular 3D box or a cube or a sphere.

[0229] In another embodiment of the present invention, the data stored in the database system includes one or more task instructions associated with corresponding equipment.

[0230] Figure 2 An exemplary task instruction 200 (i.e., an exemplary AR object viewed by a user through AR glasses) according to an embodiment of the present invention is shown. The task instruction AR object 200 represents the task instruction "Start dosing line 1" (A2), which is a task that an operator performs through Figure 5 A portion of the AR glasses depicted in FIG.

[0231] The task instructions visualized (i.e., displayed) by the AR glasses and superimposed on the view of the factory seen by the user through the AR glasses include, but are not limited to, one or more of a short text 202, a detailed text 204, a photo 206, and / or a video icon 208. In one embodiment, the program logic 106 associates the spatial coordinates (defined relative to the real-world coordinate system) of the task instructions stored in the database and linked to (i.e., corresponding to) the corresponding equipment with the spatial coordinate system used by the AR glasses to display virtual objects (hereinafter referred to as the "AR glasses coordinate system"), and anchors the task instructions to the spatial coordinates defined relative to the coordinate system. In one embodiment, the AR glasses coordinate system to which the task instructions are anchored is an anchor point of a grid network covering (i.e., defining) the corresponding equipment, and the program logic 106 continuously updates the coordinates of the virtual objects within the AR glasses coordinate system so that the display of the task instructions does not move when the user of the AR glasses moves or changes the viewing direction.

[0232] In another embodiment, the content of the task instruction includes text, photos, and videos, which provide information about the corresponding equipment, such as operating parameters, including, for example, maximum and minimum values ​​of the operating parameters, maintenance requirements and / or history, raw materials used and products produced, photos of the corresponding equipment corresponding to the task instruction, etc. In one embodiment, if the task instruction includes a video icon, the user can start the video by pointing a finger or an object such as a pointer to the display of the video icon as seen through the AR glasses, and the program logic 106 executes (i.e., starts) the video in response to detecting the pointer or finger. In one embodiment, the pointer or finger is partially covered with a material that reflects a specific wavelength (such as a narrowband wavelength), and the program logic is configured to start the execution of the video for a minimum amount of time when the wavelength is received. For example, if the pointer is positioned at the video icon for at least 3 seconds, the video is executed by the program logic 106.

[0233] In another embodiment, the equipment includes machines (e.g., machines that process raw materials or intermediate products, such as pumps, mixers, material conveyors (i.e., conveyor belts, pipes, etc.), ovens, automation systems or parts of automation systems, robots and / or robotic systems, spectrometers, chromatographs, etc.) as well as non-participating factory equipment, such as floors, walls, ceilings, room units, racks, supports, etc.

[0234] Return to reference Figure 1A , the automation system further includes a distributed control system 132 coupled to the database system 102. The distributed control system (DCS) 132 is also coupled to the sensors 112, 114, 116, the actuators 118, 120, the equipment 122, and the robot 124, and is configured to control the operation of the sensors 112, 114, 116, the actuators 118, 120, the equipment 122, and / or the robot 124 and / or receive information from the sensors 112, 114, 116. In one embodiment, the DCS 132 includes: a memory 134 configured to store one or more process parameters corresponding to the equipment 122 and / or the robot 124 and process control software 136; and a processor 138 configured to execute the process control software 136 to automatically control the production process in the factory. For example, the processor 138 may be configured to execute process control software for automatically controlling a production process based on measurement signals received from one or more sensors 112 , 114 , 116 and control signals sent to one or more actuators 118 , 120 .

[0235] Process parameters indicate the current state of a production process of a plant, such as operating parameters of equipment, including the state of the equipment (eg, an indication of a fault) and values ​​of various equipment parameters, such as flow rate, pressure, temperature, and the like.

[0236] The sensors are configured to measure one or more process parameters of the equipment or parameters of the plant itself, such as ambient environment parameters, and the actuators are coupled to one or more equipment 122 and / or robots 124 for controlling the process parameters of the equipment and / or robots.

[0237] In another embodiment, the automation system also includes a user database 140 coupled to the DCS 132. The user database 140 is configured to store user information associated with the production process. For example, in one embodiment, the user information includes a user ID, a user role, and / or user permissions. In one embodiment, the program logic 106 controls the content of the AR visualization (i.e., which task instructions (if any) are to be displayed for a particular user, and / or what content of each task instruction is to be displayed for a particular user) based not only on the spatial location of the user but also on the user information (corresponding to a particular user) received by the program logic 106 from the user database 140 to be displayed in the AR glasses for a particular user of the AR glasses. That is, the program logic 106 "filters" the received task instructions and / or the content of the task instructions based on the user of the glasses, as indicated by the user information received from the user database 140.

[0238] Thus, in one embodiment of the invention, executable program logic 106 is configured to receive user credentials via, for example, a user interface of AR glasses (not shown), or alternatively via a smartphone, other portable device, or via a scanning device that scans a potential user's personal features (such as an image of a retina or finger). The user credentials may be entered into the automated system by the user using, for example, a keyboard or voice command via a wireless connection or a wired connection, or the user credentials may be received by the automated system via a scanning device. Upon receiving the user credentials, program logic 106 authenticates the user based on matching the received user credentials with, for example, a user ID in the user ID.

[0239] Figure 3 An exemplary node 300 of a graph database is shown, which represents a task instruction, which can be graphically represented as follows: Figure 2 The AR object shown.

[0240] Figure 4 A flow chart of a method for operating an automation system is shown. For example, the method may be based on Figure 1AThe method includes providing an automated system step 402. The automated system includes an augmented reality system, the augmented reality system including: a database system 102 having stored data and spatial coordinates associated with corresponding equipment in a plurality of equipment, augmented reality glasses 104 configured to display data, and executable program logic 106 operably coupled to the augmented reality glasses, as shown in, for example, Figure 1A The augmented reality system may include additional components, such as a control system 132 and / or one or more devices.

[0241] Next, in step 404, the executable program logic 106 receives the latest spatial position of the user wearing the AR glasses from the positioning system 108. The position information can be obtained by Figure 1A In other embodiments, the program logic 106 and the positioning system 108 may be integrated components of the AR glasses and may receive the user's location information via a local interface (eg, a bus).

[0242] Next, in step 406, the database system determines the data of the equipment that is close to the latest spatial position of the user wearing the glasses. For example, this determination can be performed automatically or in response to a request from program logic 106.

[0243] Next, in step 408, the program logic 106 receives data of the determined equipment from the database system. For example, this can be performed in response to submitting a request to the database system or based on the subscription of the program logic 106 to a messaging service provided by the streaming interface of the database system 102. The submitted request and / or subscription includes the latest location information of the user wearing the glasses, so that the database system can identify the equipment in the equipment that is spatially close enough to the user wearing the AR glasses in step 406.

[0244] Next, in step 410, the executable program logic 106 causes the AR glasses to display at least some of the received data, such as a GUI with instructions for operating or maintaining the equipment. The displayed data can enable the user to control and / or maintain the equipment, and therefore also control and / or maintain the automated production process.

[0245] Figure 5 An exemplary GUI displayed to a user who has been assigned the "Operator" role is shown. For example, the operator can wear AR glasses and view real equipment. Task instructions A1-A6 are shown by the AR glasses at the relevant x / y / z coordinates. For example, the AR glasses can use the xy coordinates of different equipment (such as dosing units, conveying and heating units, or packaging units) to display corresponding operating instructions or videos at a predefined distance relative to the real world position of the real world equipment.

[0246] For example, a user wearing glasses may have a registered user account in a user database, and its user ID may be associated with one or more user roles (in this case, an "operator" role). Additionally or alternatively, each user role may optionally include one or more user permissions.

[0247] Therefore, the executable program logic 106 is configured to select the content of the received data to be displayed (e.g., "filter" the content of the received task instructions) based on one or more user roles and / or one or more user permissions associated with the user ID. For example, as a non-limiting exemplary embodiment only, the one or more user roles include an operator role, a maintenance role, an engineering design role, an administrative role, and a visitor role. The one or more user permissions and / or roles limit and determine what can be displayed in the AR world through the AR glasses. For example, in the sense of viewing the data corresponding to the equipment by displaying the corresponding task instructions, the user permissions for the engineering design role can define which equipment in the factory can be accessed by the user associated with the user ID with the engineer role. A user associated with a user ID can have more than one role and more than one user permission.

[0248] exist Figure 5 In the example shown, the user wearing the AR glasses has been assigned the role of "Operator". Therefore, the executable program logic causes the AR glasses to display view 500, in which the user sees the operating instructions A1 on how to clean the filter when approaching the dosing unit, and can further see additional task instructions and corresponding instruction videos A2, A3. When the user approaches the conveying and heating unit, task instruction A7 will be seen via the AR glasses. Similarly, when approaching the packaging equipment, task instructions A4-A6 will be seen. Tasks A1-A7 involve standard tasks that must be performed in the normal operating mode of the industrial plant.

[0249] Figure 6 An exemplary GUI 600 is shown displayed via AR glasses to a user who has been assigned the role of "maintenance worker". When the maintenance worker wears the AR glasses and views the real equipment, he will see task instructions related to maintenance tasks rather than operational tasks. For example, when approaching the dosing unit, the user will see task instructions B1 for changing a defective pressure sensor 1234, and / or when approaching the conveying and heating unit, he will see instructions B2 on how to check the heater.

[0250] Figure 7 The use of a spatial anchor is shown and an avatar of a remote user is shown.

[0251] The remote human operator 702 may be an expert in maintaining equipment (e.g., a complex machine 708). The machine and the local operator 704 are both located in an industrial plant in city B in country / region B. The remote user 702 is located in a different place, for example, in city A in country / region A. The augmented reality system 101 includes executable instructions in the form of an augmented reality application (AR application). In addition, the AR system 101 includes a virtual reality application 701 ("VR application"). The VR application is interoperable with the AR application. The AR system 101 enables remote users to support local colleagues 704 in various operations and / or maintenance tasks performed locally on or at the equipment 708 to be used. The AR system is configured to create an avatar 706 for the remote user and position the avatar at a defined location, for example, at a location close to the machine 708 where his support is needed.

[0252] According to a preferred embodiment, the AR system includes a VR application configured to generate a virtual reality for a remote user, which enables the remote user to see the local user 704 and / or other real-world objects at the industrial plant (e.g., machine 708) as perceived from the perspective of the avatar 706. For example, the remote user can see the real-time values ​​of the production line and the machine through a robot equipped with a camera and / or other sensors located at the same position and orientation as his avatar 706. In some embodiments, the robot is remotely controlled by the remote user 702 and enables the remote user to perform defined operations through the robot. The avatar has a defined position in a 3D environment generated by the AR application 106 and displayed to the local user 704 via the AR glasses 104. In addition, the local human user 704 (AR user) and the machine 708 have a defined position within the 3D environment used as an augmented reality ("mixed reality") coordinate system. The executable program logic 106, for example, implemented as an AR application, is operably coupled to the AR glasses 104 and also coupled to the VR application 701 of the remote user. The AR system 101 is configured to ensure that the VR coordinate system seen by the remote user and the AR coordinate system seen by the local user 704 via the AR glasses have the same coordinate system as the basis for positioning virtual objects (avatars, holograms, GUIs, etc.). The AR application 106 and the VR application 701 receive position information of virtual objects to be displayed in augmented / virtual reality from the same database system 102. According to some embodiments, the VR application generates a virtual reality ("manufacturing metaverse") in which the avatar of the local human operator 704 is shown in a defined optical representation (e.g., only a name or even a 3D avatar shape) so that the remote user can see the local user within the VR application (not shown). Vice versa, the local user (AR user) 704 can view the avatar 706 of the remote user 702 as a hologram (e.g., only as a name or as a 3D avatar shape) within the "manufacturing metaverse" presented to the local user 704 as augmented reality via the AR glasses 104.

[0253] According to an embodiment, the AR glasses 104 include a microphone 710 and an acoustic output interface 712, for example, a speaker that provides a user interface to the local user. Similarly, the VR application 701 may include a user interface including a microphone and an acoustic output interface for enabling a remote user 702 to interact with the VR application. The AR application 106 operably coupled to the local user's AR glasses interoperates with the VR application so that the remote user 702 and the local user 704 can talk to each other. Preferably, the AR application uses the position information of the avatars of the local user and the remote user in a shared coordinate system to control the volume of the acoustic output interface of the AR glasses: the closer the local user 704 is to the avatar 706, the louder the volume. Similarly, the sensitivity of the microphone can be adjusted according to the distance between the two users in the shared coordinate system: the greater the distance, the lower the sensitivity of the microphone.

[0254] Thus, an AR system including an AR application and a VR application according to the depicted embodiment allows two users to collaborate using a shared coordinate system also referred to as a "manufacturing metaverse." The manufacturing metaverse consists of a virtual coordinate system shared by all users and AR objects in the metaverse, whereby the virtual coordinate system is mapped to and presented as an overlay of real-world objects 708 of an industrial plant. This enables an experienced remote user to experience the same or similar visual context as the guided local user.

[0255] According to the first scenario, thermoplastic polyurethane (TPU) production is carried out at location B. The production line is modified to adapt it to new products that have never been produced by the production line before. The remote operator is an expert in this type of production and lives in location A (country / region A, city A). The remote user will guide the local operator when and which valves should be checked. The remote user will also inform the local user of important things that the local user should pay attention to in order to ensure that the production is running smoothly. In addition, the remote user can guide the local user to perform the right operating steps in the right situation and at the right time.

[0256] According to the second scenario, TPU production is already underway at location B. For the night shift, the work coordinator for that night shift is sick. An experienced human operator at remote location A (other time zone, country A, city A) can support the production at location B during the day shift. He will use his avatar to collaborate directly with one or more local human operators. The remote user can coordinate the work of the local workers by creating processes / tasks that can be performed by the local user (AR user).

[0257] According to the third scenario, autonomous TPU production is taking place at location B. Under normal / standard conditions, the production line can run completely autonomously. Manual operation is only required when unforeseen circumstances occur. This supervision will be done through the manufacturing metaverse using a VR application from a remote location. One or more robots are coordinated and triggered by a remote human operator by creating / using the correct process / task for the robot. In the case where the remote human operator needs to see a real picture and / or video stream of the local situation, the remote user is enabled to move to the equipment of interest via a VR application and an AR application associated with the robot's control program. The robot is equipped with a camera and is controlled to take the same position and orientation as the position of the remote user's avatar. When the robot has reached the location, the robot's camera is made to take an image and / or video of the equipment in front of the avatar / robot, and the image or video is transmitted to the VR application. The remote user will see the image and / or video via the virtual reality created by the VR application. When the robot is controlled to take images and / or videos from the same position and orientation as the remote user's avatar in the mixed reality coordinate system of the industrial plant, the photos and images will show the equipment of interest from the same angle as the local human operator at the said position and orientation.

[0258] Embodiments of the present invention can be used in many other scenarios and industries. For example, embodiments of the present invention can be used in the automotive industry and allow experienced engineers to support colleagues working in other locations of the automotive manufacturing company.

[0259] Likewise, AR systems can be used in healthcare operations and allow medical specialists or doctors to support other doctors working in other hospitals.

[0260] In some use case scenarios, robots are not only used to acquire images or videos, but can also be used to solve problems under the control of a remote user. For example, robots can be used to perform maintenance tasks in locations that are dangerous for human operators, such as in the case of deep-sea construction operations, or in the case of tasks performed in environments contaminated or at risk of being contaminated by radioactive materials or toxic chemicals.

[0261] According to some embodiments, spatial anchors are used to position virtual objects at a defined position and orientation relative to real-world objects such as equipment 708. For example, it may be desirable to display a GUI hologram so that a user can monitor and control equipment 708 at a distance of approximately 40 cm in front of the equipment.

[0262] To ensure that the local user 704 will always see the GUI hologram at this defined location in the augmented reality, regardless of the current location of the user 704, the executable program logic 106 generates and displays the GUI hologram at the spatial anchor or at a defined distance relative to the spatial anchor. According to some embodiments, the spatial anchors (i.e., at least the anchor ID and the anchor coordinates) are stored in a database system so that they can be accessed by the VR application 701. The VR application is configured to read the stored spatial anchors and generate and display the same virtual object, e.g., a GUI hologram of the equipment 708, at the spatial anchor or at a defined distance relative to the spatial anchor.

[0263] According to some embodiments, spatial anchors are defined and created by placing machine-readable codes (e.g., QR codes 711, 713, 714) at various locations within an industrial plant. The AR glasses 104 may include a camera that acquires a digital image of the machine-readable code, extracts the anchor ID encoded therein, creates an anchor with the coordinates of the machine-readable code in the real-world coordinate system, and stores these anchors in a database system. Alternatively, the user 704 may create a spatial anchor by performing an anchor creation gesture at a desired location in the real world. The gesture is captured by the camera of the AR glasses, and the spatial anchor is similarly created and stored by the executable program logic 106. The anchor ID may be automatically created when the spatial anchor is created.

[0264] Figure 8 A coordinate system 800 is shown in which a VR application 701 is used to generate a virtual representation 808 of real-world equipment 708 of an industrial plant and display the virtual representation to a remote user 702 via virtual reality display technology. The virtual reality coordinate system 800 may also include a virtual representation 804, such as an avatar, of a local user 704 operating in spatial proximity to a real-world object 704. The distance of the virtual representations 808, 804 of the equipment 708 and the local user 704 within the coordinate system 800 corresponds to and reflects the real distance of the real-world object, which can be represented via a spatial grid of equipment and local users 704 covering the equipment 708 at the industrial plant in the real world. In addition, the remote user 702 can be represented in the coordinate system 800 as a virtual entity, for example, an avatar 706 that the local user 704 can see via AR glasses 704. Figure 8 A coordinate system shared by the virtual reality generated by the VR application and the AR reality generated by the AR application is shown, but preferably the remote user sees objects and avatars in coordinate system 800 from the perspective (position and orientation) of his avatar 706 (not shown).

[0265] Fig. 9A flow chart showing a method for operating an automation system for controlling an automated production process of an industrial plant is shown. For example, the method may be based on Figure 1A The automated system includes an augmented reality system, the augmented reality system including augmented reality glasses 104 configured to display data and an executable program logic 106 operably coupled to the augmented reality glasses, as shown in FIG. Figure 1A The augmented reality system may further include a database system 102 having stored data and spatial coordinates associated with respective equipment in the plurality of equipments. The augmented reality system may further include additional components, such as a control system 132 and / or one or more equipments in the equipments.

[0266] In step 902, the executable program logic receives the latest spatial location of the user wearing the augmented reality glasses from the positioning system 108. The location information can be obtained via Figure 1A In other embodiments, the program logic 106 and the positioning system 108 may be integrated components of the AR glasses and may receive the user's location information via a local interface (eg, a bus).

[0267] In step 904, the executable program logic determines the coordinates of the bounding space near the user wearing the AR glasses. The executable program logic 106 is configured to determine the coordinates of the bounding space near the user based on one or more of the following bounding space parameters: the user's most recent spatial position, the user's line of sight, the number of equipment near the user or in the user's line of sight (e.g., the density of equipment near the user and / or in the user's line of sight), the number and / or data type of virtual objects corresponding to equipment near the user and / or in the user's line of sight, the configuration of the walls, floors and / or ceilings of the factory infrastructure near the user, and one or more user roles assigned to the user wearing the AR glasses 104. The bounding space is thus defined by the coordinates of the bounding surface that defines the bounding space, as determined by the program logic. The program logic can also be configured to interpolate between the determined coordinates to add more coordinates that define the bounding surface of the bounding space at a higher resolution.

[0268] In step 906, the executable program logic displays one or more virtual objects to the wearer of the AR glasses via the AR glasses if and only if the coordinates of the virtual objects are within the boundary space. The displayed virtual objects include data associated with monitoring and / or controlling the automated production process. The displayed virtual objects include data, such as text data, video data, image data, audio data, or alarm data. For example, in one embodiment, the program logic sends the latest spatial position of the user wearing the AR glasses and the coordinates of the boundary space (e.g., the coordinates of the boundary surface surrounding the boundary space) to the database system 102, and receives from the database system those virtual objects with coordinates contained in the boundary space defined by the coordinates of the boundary space. Therefore, the user of the AR glasses (e.g., the wearer) views the overlay of the virtual objects contained in the boundary space with the real world objects of the factory. Then, the user can use the text data, image data, video data, audio data, and / or alarm data corresponding to the displayed virtual objects to maintain and / or control the corresponding equipment corresponding to the virtual objects, and / or obtain the operating status, parameter settings, and / or educate themselves on how the equipment operates and / or the roles of different equipment in the production process.

[0269] In step 908, the executable program logic continuously updates the three-dimensional bounding space and the coordinates of the virtual objects displayed therein according to the latest spatial position of the user wearing the AR glasses. In addition, the program logic may also be configured to receive data indicating the orientation of the AR glasses and / or the orientation of the eyes, and use the additional data to determine and update the coordinates of the bounding space according to the additional data. Therefore, when the user wearing or operating the AR glasses moves to a different spatial position, and / or when the user (although stationary relative to the spatial position) changes the line of sight through the glasses by changing the orientation of the AR glasses (for example, the user wearing the AR glasses as a headgear moves the headgear by moving the user's head), the program logic determines a new bounding space. Then, the program logic 106 displays one or more virtual objects to the wearer of the glasses via the AR glasses 104 if and only if the coordinates of the virtual objects are within the new bounding space. The virtual objects displayed in the new (i.e., updated) bounding space may include some of the virtual objects displayed in the previous bounding space, depending on any overlap between the new bounding space and the previous bounding space.

[0270] According to yet another embodiment, and referring back to Figure 1A, the augmented reality system includes AR glasses 104 and program logic 106, and the database system 102 is a component of the distributed control system 132 or an independent object or component of a plant communicatively coupled to the augmented reality system and / or the distributed control system 132. In this embodiment, upon receiving the spatial position of a user wearing the AR glasses 104 from the positioning system 108, the executable program logic 106 is configured to display, via the augmented reality glasses, one or more virtual objects associated with one or more equipment near the spatial position of the user, wherein at least one of the virtual objects is linked to at least one corresponding secondary virtual object.

[0271] In one embodiment, the secondary virtual objects are stored in the database system 102 or alternatively in a database external to the AR system 101. In one embodiment, the secondary virtual objects are not currently displayed to the user via augmented reality glasses. The database system 102 includes all of the features described in the embodiments disclosed so far. For example, the database system 102 is configured to store virtual objects representing data, mappings (i.e., links) between one or more virtual objects and one or more secondary virtual objects, mappings (i.e., links) between one or more virtual objects and corresponding equipment in a plurality of equipment, and / or coordinates of virtual objects and coordinates associated with corresponding equipment in a plurality of equipment.

[0272] The executable program logic 106 is configured to receive a signal from the augmented reality glasses indicating a user's hands-free selection of a virtual object linked to a corresponding secondary virtual object. For the purposes of this disclosure, "hands-free" selection of a virtual object viewed by a user via the AR glasses includes any manner in which the user does not use their hands to select a virtual object. That is, the user does not use their hands to select a virtual object, which is particularly advantageous, for example, when the user performs maintenance on or controls equipment, or when the user's hands are otherwise busy performing various tasks or holding various items.

[0273] In one embodiment, displaying a virtual object to a user via AR glasses linked to a secondary virtual object (also referred to as a linked virtual object or an optional virtual object) is different from displaying a virtual object to a user via AR glasses that are not linked to a secondary virtual object (also referred to as a non-linked virtual object). In one embodiment, a linked virtual object may be displayed together with a limited amount of information (such as a limited amount of text data and / or image data). Although according to some embodiments, a linked virtual object is not "clickable" in the conventional sense that a user moves a cursor over highlighted text via the physical movement of the user's finger and / or hand, a linked virtual object is similar to a conventional "clickable" link in the sense that once selected, the user views additional information associated with the link (e.g., selection of a conventional clickable link directs the user's Internet browser to a different website address (i.e., a different URL) for displaying additional information associated with a clickable link). According to an embodiment, the display of a linked virtual object may be different from the display of a non-linked virtual object in terms of, for example, the size of the display of the linked virtual object, the highlighting of the displayed text / image of the linked virtual object, the color of the display of the linked virtual object, the number of displayed linked virtual objects, and / or a combination of the color and number of displayed linked virtual objects.

[0274] In one embodiment, the database system 102 is additionally configured to store secondary virtual objects and a mapping (ie, link) between each secondary virtual object and a corresponding linked virtual object (ie, store virtual objects and corresponding corresponding secondary virtual objects).

[0275] The executable program logic 106 is also configured to display, via the augmented reality glasses, a corresponding secondary virtual object linked to the virtual object selected by the user hands-free. In one embodiment, the signal received by the executable program logic 106 identifies the selected virtual object, for example, via the coordinates of the selected virtual object. The executable program logic 106 then accesses the database system 102 to obtain the corresponding secondary virtual object associated with the selected virtual object. In one embodiment, the AR glasses 104 and / or the positioning system 108 are configured to generate a signal based on the user's hands-free selection of the virtual object, as discussed in more detail further below.

[0276] In addition to or in lieu of displaying a corresponding secondary virtual object linked to a virtual object selected by the user hands-free via the augmented reality glasses, the executable program logic 106 is configured to trigger an action associated with the selected virtual object. For example, the triggering action includes one or more actions such as: sending a command to one of the equipment associated with the selected virtual object, wherein the command causes the equipment to start, stop, or change its operation; activating a command interpreter for voice commands, whereby the command interpreter is configured to process and interpret voice commands for controlling the equipment associated with the selected virtual object according to a command list specified in the selected virtual object; and / or disabling at least one other command interpreter, which is configured to process and interpret voice commands according to a command list specified in an unselected virtual object.

[0277] For example, the voice detection unit may comprise different command interpreters for different equipment and respectively assigned virtual objects. The selection of one of the virtual objects may result in the activation of the command interpreter associated with the selected virtual object and / or the deactivation of other command interpreters. This may have the advantage that a production line may be easily replenished with more equipment and corresponding command interpreters without running the risk of conflicts that may arise if, for example, the command "start" would activate tasks performed by a plurality of different equipment. In addition, this feature may keep the voice commands simple, since, depending on the currently selected virtual object, the command "start" may only trigger a start action in the equipment represented by the selected virtual object.

[0278] According to one embodiment, the virtual object represents data that enables a user to operate and / or maintain multiple equipment. Additionally, the secondary virtual object linked to the corresponding virtual object represents the data associated with the corresponding virtual object. For example, in one embodiment, if the displayed virtual object identifies itself as having a specific data type (e.g., text data, image data, video data, audio data, alarm data) to the user (e.g., via a specific color, color plus number, size and / or highlighting (e.g., highlighted portion, highlighted text, highlighted image)), the corresponding secondary virtual object linked to the displayed virtual object represents the same type of data. As an exemplary, non-limiting example, the virtual object can be displayed as a geometric figure (such as a square, rectangle, ellipse or any polygon) or a GUI (e.g., with a specific shape, text block and / or data input field).

[0279] After dynamically assigning a color to a virtual object, the entire virtual object may have a specific color, or the specific color may be displayed as a background color or a colored border of a polygon or GUI without the virtual object having any other identifying features.

[0280] Thus, the secondary virtual object linked to the corresponding virtual object represents (e.g., contains) textual data, such as user instructions, maintenance instructions, production information, operating status, and / or operating parameters associated with the equipment to which the corresponding virtual object corresponds. For example, when the corresponding virtual object is displayed to the user via AR glasses near the equipment, the corresponding virtual object corresponds to the equipment.

[0281] In one embodiment, the executable program logic is configured to: determine the number of virtual objects linked to corresponding secondary virtual objects (i.e., linked virtual objects) associated with one or more equipment near the spatial location of the user; dynamically assign a different color from a predefined color set of colors to each virtual object linked to the corresponding secondary virtual object before the virtual object is actually displayed to the user via the AR glasses and display each virtual object linked to the corresponding secondary virtual object with the dynamically assigned different color. In addition, if the number of virtual objects linked to the corresponding secondary virtual object to be displayed to the user is greater than the number of colors in the predefined color set, the executable program logic is configured to display only those virtual objects linked to the corresponding secondary virtual object that are assigned different colors. By displaying only those linked virtual objects having different colors such as selected from the predefined color set of different colors, each colored linked virtual object has particularly unique characteristics, thereby enabling the user of the AR glasses to more easily select any particular colored linked virtual object via hands-free operation.

[0282] According to another embodiment, when the number of virtual objects linked to a corresponding secondary virtual object is greater than the number of colors in a predefined color set, the executable program logic is configured to select those virtual objects assigned unique colors by using a predefined priority ranking of linked virtual objects (also referred to as selectable virtual objects).

[0283] For example, and as discussed in further detail below, the predefined priority ordering of selectable virtual objects may be based on the data type of the virtual objects. That is, and by way of non-limiting illustrative example only, a virtual object that references alarm data may have a higher priority than a virtual object that references image data, which may have a higher priority than a virtual object that references audio data, which may have a higher priority than a virtual object that references video data. Thus, if the predefined color set has only three colors, such as red, blue, and yellow, and if the field of view of the AR glasses includes equipment with seven associated selectable virtual objects, one of which represents alarm data, four of which represent image data, one of which represents audio data, and one of which represents video data, then executable program logic may display a virtual object representing alarm data in a first distinct color selected from the three distinct colors in the predefined color set, display one of the four virtual objects representing image data in a second different distinct color selected from the three distinct colors in the predefined color set, and display a virtual object representing audio data in a third distinct color selected from the three distinct colors in the predefined color set. The other three virtual objects representing image data and the virtual object representing video data will not be displayed. In another embodiment, if a color is to be assigned to a selectable virtual object in a group of two or more selectable virtual objects that reference the same data type, the program logic randomly selects a virtual object from the group of two or more virtual objects, assigns a different color to the selected virtual object, and only displays the selected virtual object with the assigned color.

[0284] In another embodiment, when preconfiguring the AR system or initiating a user session with the AR system, the user may specify a color set having two or more distinct colors, and program logic may store these color sets in a retrievable manner in a table, such as in local memory or database system 102.

[0285] Fig.10A portion of a factory 1000 is shown with equipment 1002, 1004, 1006, and 1008 located near a user 1010 (e.g., the user's 1010's most recent spatial location). In one embodiment, equipment near the user's spatial location includes equipment within the field of view of the user wearing the AR glasses, or within the field of view limited by a maximum distance "r" from the user as defined or predefined by the user, or within a bounding space near the user, or equipment located near the user's spatial location, as well as other equipment that is not necessarily located near the user, but is included when the role of the user of the AR glasses enables the user to view virtual objects associated with equipment located in different rooms and / or on different floors of the factory. The field of view of the user wearing the AR glasses, the user-defined or predefined maximum distance from the user, the bounding space near the user, and the role of the user in filtering equipment that is considered to be located "close" to the user's spatial location have been previously discussed in this application and are described in detail below. Fig.10 within the scope of this embodiment.

[0286] like Fig.10 As shown in the embodiment of FIG. 1 , the equipment close to the user 1010 is the equipment located within the field of view 1012 of the AR glasses (not shown) worn by the user 1010 and within a maximum radius r from the user. The scope of this embodiment covers a radius r defining a circle, hemisphere, or sphere surrounding the user 1010, or a portion of a sphere (e.g., a bounding volume or bounding space) defined by a portion of the outer boundary surface of the sphere of radius r, the outer boundary surface having a boundary surface side of radius r, whose vertex is centered at the user 1010. For ease of illustration and explanation, Fig.10 Equipment is shown as being within the field of view 1012 of AR glasses worn by a user 1010 and within a maximum radius r from the user, which is defined as equipment close to the user 1010, however, the scope of this embodiment covers equipment that is spatially close to the user, such as equipment located within a 2D or 3D bounding space of any shape and size as previously discussed in conjunction with other embodiments. Fig.10 Also shown is equipment 1014 having a portion 1016 positioned proximate to the user and a portion 1018 not positioned proximate to the user. Additionally, equipment 1002 is associated with two linked (i.e., optional) virtual objects 1020, 1022, equipment 1006 is associated with a linked (i.e., optional) virtual object 1024, and equipment 1008 is associated with a linked (i.e., optional) virtual object 1026.

[0287] According to another embodiment, when the number of virtual objects linked to the corresponding secondary virtual objects to be displayed to the user is greater than the number of colors in the predetermined color set, the executable program logic is configured to assign colors to the virtual objects and only display those virtual objects that are assigned unique colors, thereby performing color assignment according to the distance sorting of the linked virtual objects, for example. For example, in one embodiment, the executable program logic is configured to dynamically determine the distance from the user to each equipment in the equipment with the corresponding optional virtual object, and assign display priority to the corresponding optional virtual object based on the determined distance. Therefore, in the case where the predefined color set includes four colors "red", "green", "blue" and "yellow", and the number of virtual objects in the user's current boundary space includes five virtual objects, these four colors are assigned to the virtual objects according to the distance of the virtual object to the user. The virtual object farthest from the user in the boundary space will not be assigned any color, because the virtual object to be displayed is more than the color present in the predefined color set. As a result, the virtual object farthest from the user in the boundary space will not be displayed, and therefore will not be selected via a voice command indicating a color in the predefined colors.

[0288] According to an embodiment, the creation of a predefined color set includes creating a mapping, for example, a mapping file or a mapping entry in a database, whereby the mapping assigns each of the predefined colors to a voice command indicating the corresponding color. The creation of a predefined color set may be performed, for example, at the initialization or instantiation of an executable program logic or at the login of a user 1010. When a color is dynamically assigned to a virtual object to be displayed, the mapping is updated so that an identifier of the virtual object (or an ID of an equipment represented by the virtual object) is stored in association with the color assigned to the virtual object, thereby enabling the executable program logic to identify the virtual object selected via a voice command by analyzing the mapping. When the user 1010 and / or the equipment to which the virtual object representing the equipment can be anchored changes position relative to each other, the mapping is updated.

[0289] For example, if the user moves such that one or more additional virtual objects are now contained within the bounding space and / or such that one or more of the previously displayed virtual objects are now outside the bounding space, the colors are reassigned and, if necessary, additional virtual objects are displayed, provided that each virtual object to be displayed is assigned one of the colors in the predefined color set and / or the previously displayed objects are no longer displayed.

[0290] In an embodiment, the executable program logic is configured to assign a higher priority to the display of corresponding optional virtual objects for a smaller determined distance. For example, in an embodiment and as illustrated, the program logic assigns the highest display priority to virtual object 1024, the next highest display priority to virtual object 1026, and the lowest display priorities to virtual objects 1020 and 1022 based on r2 < r3 < r4. Thus, if the predefined set of colors has only two distinct colors, the executable program logic assigns the first distinct color to virtual object 1024, the second distinct color to virtual object 1026, and displays only optional virtual objects 1024 and 1026. Optional virtual objects 1020 and 1022 are not displayed. However, for example, if the predefined set of colors has three distinct colors, the executable program logic assigns the first distinct color to virtual object 1024, the second distinct color to virtual object 1026, and the third distinct color to one of virtual objects 1020 and 1022, and then displays only optional virtual objects 1024, 1026, and one optional virtual object among virtual objects 1020 and 1022 that has been assigned a distinct color.

[0291] When the number of virtual objects linked to a corresponding secondary virtual object is less than or equal to the number of colors in the predefined set of colors, the executable program logic is configured to assign a distinct color selected from the predefined set of colors to each optional virtual object and display each colored virtual object among the selectable distinct-color virtual objects to the user via the AR glasses.

[0292] When the user changes their spatial position and / or when the line of sight of the AR glasses changes due to, for example, movement of the head, and / or when the eye orientation through which the user views via the AR glasses changes, then some of the currently displayed optional colored virtual objects disappear from the field of view through the AR glasses, and new virtual objects associated with new equipment newly approaching the user are assigned colors by the program logic, where the new colors are selected from the colors previously assigned to virtual objects that are no longer close to the user (i.e., have disappeared from the field of view) and / or other unused colors still available from the predefined set of colors. The program logic continuously updates which optional virtual objects associated with equipment approaching the user are assigned colors and displayed based on the updated distance of the corresponding equipment from the user's spatial position, and these optional virtual objects also continuously change due to user movement (i.e., body, head, and / or eye movement).

[0293] The program logic advantageously updates the optional virtual objects to be assigned distinct colors and displayed to the user based on the dynamically changing distance of the corresponding equipment from the user's spatial position, thereby showing the optional virtual objects only based on their proximity relevance to the user without overwhelming the user with too much information.

[0294] In another embodiment, the executable program logic is configured to dynamically assign a color to each virtual object linked to the corresponding secondary virtual object and to display each virtual object linked to the corresponding secondary virtual object in the dynamically assigned color.

[0295] In still other embodiments, the executable program logic displays only those virtual objects linked to the corresponding secondary virtual objects in dynamically assigned colors that are associated with equipment that is close to the user's spatial location, which is the user's current latest spatial location. For example, equipment close to the user's spatial location includes equipment within the field of view of the user wearing the AR glasses, or within the field of view limited by a maximum distance "r" from the user as defined or predefined by the user, or within a boundary space close to the user, or equipment located near the user's spatial location, and other equipment that is not necessarily located near the user but is included when the role of the user of the AR glasses enables the user to view virtual objects associated with equipment located in different rooms and / or on different floors of the factory. The field of view of the user wearing the AR glasses, the user-defined or predefined maximum distance from the user, the boundary space close to the user, and the role of the user in filtering equipment that is considered to be located "close" to the user's spatial location have been previously discussed in this application.

[0296] In another embodiment, the executable program logic is configured to dynamically assign colors to each virtual object displayed to the user via AR glasses linked to the corresponding secondary virtual object based on the data type or equipment associated with each virtual object. As briefly discussed above, colors can be assigned to virtual objects according to the data type represented by the virtual object. In one embodiment, the executable program logic is configured to dynamically assign one of the primary colors and / or secondary colors and / or tertiary colors to each virtual object linked to the corresponding secondary virtual object, wherein each virtual object is assigned a different color. The primary colors are red, yellow and blue, the secondary colors are obtained by mixing two primary colors and include orange (red + yellow), green (blue + yellow) and purple (red + blue), and the tertiary colors are obtained by mixing the primary colors and the secondary colors and include vermilion (red + orange), amber (yellow + orange), yellow-green (yellow + green), cyan (blue + green), violet (blue + purple) and magenta (red + purple). In other embodiments, the executable program logic is configured to dynamically assign one of red, green, blue and yellow, and optionally include black and white. By dynamically assigning colors as described above, in particular primary colors, or one or more secondary colors of primary colors and secondary colors, and optionally black and / or white, when the augmented reality system is implemented as a hands-free voice selection of colored virtual objects, as described in more detail below, the augmented reality system advantageously does not have to train the user's voice, because selecting a specific virtual object by speaking the color of the virtual object only involves simple single-word recognition. In addition, in another embodiment, the user speaks the name of the color of the virtual object in English. Therefore, the augmented reality system can be advantageously used by users of different native languages ​​other than English around the world, because colors are universally recognizable, and most non-native English speakers at least know the English words for the basic colors (e.g., primary colors plus one or more secondary colors) plus "black" and "white", or can easily learn the English names of these colors due to simple pronunciation and / or ease of use of translation software.

[0297] As an exemplary non-limiting embodiment, the executable program logic dynamically assigns colors, such as blue, green, red, and yellow, where blue is predefined to be assigned to a virtual object representing text data, green is predefined to be assigned to a virtual object representing image data, yellow is predefined to be assigned to a virtual object representing video data, and red is predefined to be assigned to a virtual object representing text data. For example, when the displayed linked virtual object changes due to movement of the user (e.g., movement of the head of the user wearing the AR glasses from a first line of sight to a second line of sight, movement of the eyes of the user wearing the AR glasses from a first eye orientation to a second eye orientation, and / or movement of the user's spatial position from a first spatial position to a second spatial position), the executable program logic dynamically assigns a color to the virtual object newly displayed to the user via the AR glasses based on the data type of the newly displayed virtual object.

[0298] In another embodiment, if the two or more virtual objects represent the same data type (e.g., video data), the executable program logic assigns a yellow color to each of the two or more virtual objects. The AR glasses are configured to recognize hands-free voice selection of colored virtual objects by speaking the color of the virtual object in combination with the position of the displayed virtual object in the AR glasses (as seen by the user), such as speaking "right yellow" to voice select the yellow virtual object located on the far right in the AR glasses, speaking "left yellow" to voice select the yellow virtual object located on the far left in the AR glasses, speaking "middle yellow" to voice select the yellow virtual object located between the "left yellow" virtual object and the "right yellow" virtual object, speaking "middle right yellow" to voice select the yellow virtual object located between the "middle yellow" virtual object and the "right yellow" virtual object, and speaking "middle left yellow" to voice select the yellow virtual object located between the "middle yellow" virtual object and the "left yellow" virtual object.

[0299] In another embodiment, the executable program logic is configured to dynamically assign a color to each virtual object displayed to the user via AR glasses linked to the corresponding secondary virtual object based on the corresponding equipment associated with each virtual object. For example, the executable program logic is configured to dynamically perform the following assignments: assigning a first color to each virtual object associated with a corresponding equipment close to the user and displayed near or on the corresponding equipment (displayed to the user via AR glasses linked to the corresponding secondary virtual object); assigning a second color to each virtual object associated with a second corresponding equipment close to the user and displayed near or on the second corresponding equipment (displayed to the user via AR glasses linked to the corresponding secondary virtual object); and assigning a third color, a fourth color, ..., and an nth color to a third set of virtual objects, a fourth set of virtual objects, ..., and an nth set of virtual objects in a set of virtual objects associated with a corresponding third equipment, a fourth equipment, ..., and an nth equipment close to the user.

[0300] This may be a good heuristic to ensure that virtual objects that are close to the user in the context of an automated production line will be assigned different colors. For example, in some embodiments, each production line may include a set of multiple different equipment, and a manufacturing plant may include multiple production lines with the same structure and components. In this case, assigning different colors to different equipment types may have ensured that virtual objects to be displayed in the field of view / boundary space will have different colors.

[0301] In another embodiment, if more than one virtual object (displayed to the user via AR glasses linked to corresponding secondary virtual objects) is associated with corresponding equipment near the user, the executable program logic is configured to dynamically assign a combination of color and different numbers to each of two or more virtual objects associated with the corresponding equipment. For example, three virtual objects displayed to the user via AR glasses and associated with a first corresponding equipment are assigned a color of red, wherein the first red virtual object is assigned a number of 1, the second red virtual object is assigned a number of 2, and the third red virtual object is assigned a number of 3, and two virtual objects displayed to the user via AR glasses and associated with a second corresponding equipment are assigned a color of green, wherein the first green virtual object is assigned a number of 1, and the second green virtual object is assigned a number of 2. The AR glasses are configured to recognize hands-free voice selections of colored numbered virtual objects by speaking the color and number of the virtual objects as seen by the user via the AR glasses (e.g., speaking “red two” to voice select the red virtual object labeled 2 and speaking “green one” to voice select the green virtual object labeled 1) via voice recognition software (e.g., any commercially available voice recognition application) stored, for example, in a local memory of the AR glasses and executed by one or processing units of the AR glasses.

[0302] In another embodiment, the hands-free selection includes voice selection of a virtual object (i.e., a linked virtual object). For example, as discussed above, if the user of the AR glasses wishes to select a yellow linked virtual object, the user need only speak the word "yellow". The user of the AR glasses speaks the color of the linked virtual object the user wishes to select. The user may wish to select any particular linked virtual object to obtain more information related to the linked virtual object via subsequent display of a corresponding secondary virtual object linked to the selected virtual object.

[0303] In one embodiment, the augmented reality glasses and / or the positioning system include a voice detection unit configured to detect voice selections and convert the voice selections into signals indicating a user's hands-free selection of a virtual object. The voice detection unit includes one or more acoustic sensors and voice recognition software, such as any commercially available voice recognition software application. The voice recognition software may be stored in a local memory of the AR glasses and / or the positioning system and executed by one or a processing unit of the AR glasses and / or the positioning system.

[0304] In another embodiment, the user of the AR glasses expresses the voice selection in English. However, the scope of the present disclosure includes additional and / or alternative embodiments in which the voice recognition software recognizes two or more additional languages ​​such as (as non-limiting examples) German, French, Spanish, Italian, Russian, Japanese, and / or Korean, and translates the color selected via the voice selection into a language other than English and converts the voice selection into a signal indicating the user's hands-free selection of a linked virtual object.

[0305] In yet another additional embodiment, the hands-free selection includes a gaze selection of a virtual object linked to a corresponding secondary virtual object, also referred to as a selection of a selectable virtual object or a linked virtual object. For example, in a non-limiting exemplary embodiment, the AR glasses and / or the positioning system include one or more directional sensors configured to detect the line of sight of the AR glasses or the eye orientation of a user wearing the AR glasses. The detection of the line of sight of the AR glasses and the eye orientation of the user wearing the AR glasses has been previously discussed above in conjunction with other embodiments, which are incorporated into the present gaze selection embodiment. In addition, the AR glasses and / or the positioning system are configured via a processing unit and executable software to determine a first length of time that the line of sight of the augmented reality glasses is fixed on a selectable virtual object or a second length of time that the eye orientation of the user wearing the augmented reality glasses is fixed on a selectable virtual object, and if the first length of time or the second length of time is greater than a predefined maximum gaze duration, a signal indicating the user's hands-free selection of the selectable virtual object is generated.

[0306] Gaze selection for selectable virtual objects is advantageous in noisy factory environments and / or when non-native English speakers do not know the English words for specific colors, directions (i.e., left, right, center), or numbers (e.g., one, two, ..., nine, ten) and / or when multiple virtual objects have the same color or the same color-number combination or occupy the same relative position in the display of the AR glasses.

[0307] In another embodiment, the executable logic is configured to display the corresponding secondary virtual object via the augmented reality glasses by replacing the user's hands-free selection of the displayed virtual object linked to the corresponding secondary virtual object with the secondary virtual object. By replacing the display of the selected virtual object with the display of the corresponding secondary virtual object to which it is linked, the display of too many virtual objects is advantageously prevented, which may disrupt the view of the equipment in the factory seen through the AR glasses.

[0308] In another embodiment, at least some of the virtual objects are graphical representations of respective ones of the equipment associated with the corresponding virtual object.According to aspects of the embodiment, the graphical representation may include an image-based illustration and / or a video-based illustration of one equipment.

[0309] Fig.11 A flow chart of a method for controlling an automation system according to an embodiment of the present disclosure is shown, wherein the automation system is used to control an automated production process in an industrial plant. For example, the method may be based on Figure 1A The automation system shown in the figure is executed. The industrial plant includes a plurality of equipment 122, 124 for performing a production process. The plurality of equipment may be spatially distributed in the industrial plant. One or more of the plurality of equipment is associated with one or more corresponding virtual objects. The automation system includes an augmented reality system 101, which includes augmented reality glasses 104 and executable program logic 106 coupled to the augmented reality glasses.

[0310] In step 1102, the executable program logic receives the spatial position (e.g., the latest spatial position) of the user wearing the augmented reality glasses from the positioning system 108. The position information can be transmitted via Figure 1A In other embodiments, the program logic 106 and the positioning system 108 may be integrated components of the AR glasses and may receive the user's location information via a local interface (eg, a bus).

[0311] In step 1104, the executable program logic displays one or more virtual objects associated with one or more equipment near the spatial location of the user via the augmented reality glasses. At least one of the virtual objects is linked to at least one corresponding secondary virtual object. In one embodiment, the corresponding secondary virtual objects are stored in a database system (such as database system 102).

[0312] In step 1106 , the executable program logic receives a signal from the augmented reality glasses indicating a user's hands-free selection of a virtual object linked to a corresponding secondary virtual object.

[0313] In step 1108 , the executable program logic displays the corresponding secondary virtual object via the augmented reality glasses and / or triggers an action associated with the selected virtual object.

[0314] According to yet another embodiment, and referring back to Figure 1A, the augmented reality system 101 includes AR glasses 104, program logic 106, and a database system 102. The database system includes a task manager 142 (optionally shown). However, in another embodiment, the database system 102 is a component of the distributed control system 132 or a separate object or component of the plant that is communicatively coupled to the augmented reality system and / or the distributed control system 132.

[0315] Therefore, the database system 102 is configured to store data associated with the corresponding equipment in the multiple equipment. For example, the spatial database 110 of the database system 102 may be configured to store the data in association with the spatial coordinates of the corresponding equipment to which the data corresponds, as previously discussed in conjunction with other embodiments, which are also covered by and included in the scope of this embodiment. In one embodiment, the data includes a list of processes, each of which includes one or more tasks to be performed relative to one or more equipment. In addition, the AR glasses 104 are configured to display the data, such as the list of processes. Executable program logic 106 is coupled to the augmented reality glasses and the database system. The executable program logic is configured to receive the list of processes for display to the user of the AR glasses from the database system. At least one process in the process list is assigned to the user of the AR glasses. In addition, the executable program logic is configured to receive a selection of a process from the list of the one or more processes assigned to the user. In one embodiment, the user makes the selection via a user action, which includes actions such as hands-free user actions (e.g., gaze selection of a process (also referred to as selection of a virtual object) using the line of sight of the AR glasses or the eye orientation of the user wearing the AR glasses) and voice command selection of the process) and / or user pointing actions (e.g., selection of the process by the user pointing at the process displayed in the AR glasses via hands and / or fingers). Detection of the line of sight of the AR glasses and the eye orientation of the user wearing the AR glasses, voice commands for selecting virtual objects, and non-hands-free user actions have been previously discussed above in conjunction with other embodiments and will not be discussed in further detail. Hands-free gaze selection and voice selection and non-hands-free pointing actions are thus incorporated into the present embodiment.

[0316] In one embodiment, when the user activates the AR glasses, it may be based on the user being identified or authenticated by the AR glasses (i.e., by the AR processor, AR application, and / or AR operating system) or by program logic 106, the program logic retrieves or receives a list of processes from the database system 102 (e.g., from the database 110) based on the user identification, and in another embodiment, retrieves or receives the list of processes when and only when the user is authenticated, as discussed more fully further below. In one embodiment, the list of processes includes only those processes assigned to the user. Each process may have a status indicating to the user that the corresponding process is "open" or "in progress." An "open" process is a process with tasks, none of which have been completed, while an "in progress" process is a process in which at least one task but not all tasks have been completed. In another embodiment, the program logic retrieves or receives a list of all processes with a status of "open" or "in progress" from the database system 102 associated with the specific industrial plant in which the augmented reality system is operating. That is, all processes (whether or not assigned to the current user (i.e., current operator) of the AR glasses) are displayed to the user via the AR glasses. In one embodiment, although the user can see all processes, whether assigned to the user or not, the AR glasses are configured to only receive selections of processes that are assigned to the user of the AR glasses.

[0317] In addition, upon receiving the selection of the displayed process assigned to the user, the program logic sends the selected process to the database system. The program logic then receives additional data corresponding to the selected process for display to the user of the AR glasses from the database system. In one embodiment, the task manager 142 of the database system 102 processes the selected process and accesses the additional data (corresponding to the selected process) stored in the database system (e.g., stored in the database 110), or directly accesses the additional data (corresponding to the selected process) based on data stored in its queue or memory (not shown), and sends the additional data to the executable program logic 106 of the AR glasses.

[0318] In one embodiment, the task manager 142 assigns a process in the process list to one or more users. For example, the task manager may assign a corresponding process to be performed in an industrial plant in which the augmented reality system is operating to one or more corresponding users of the AR glasses. In another embodiment, the task manager dynamically updates the parameters associated with each process. Fig.12 The parameters associated with each process are discussed further.

[0319] The scope of this embodiment covers conventional, free or commercially available task managers. The task manager provides task management, which includes the process of managing processes and / or tasks during their life cycle. According to an embodiment of the present disclosure, task management includes one or more of the processes of assigning processes, tracking processes, updating processes, and reporting processes and / or one or more corresponding tasks associated with each process. Since the task manager 142 receives information from one or more executable program logics 106 associated with one or more corresponding AR glasses 104, and from the distributed control system 132, the information includes control and sensor data information related to factory equipment 122 and robot units 124, the task manager 142 advantageously helps: for example, assigning processes and / or tasks of processes, updating the status of each process and / or the tasks of each process, setting process and / or task priorities, and managing the permissions of users to perform specific processes and / or tasks. The task manager 142 provides processes, and task management can be used as the basis for efficient production processes in industrial plants, including maintaining equipment, checking for possible equipment failures, and even training personnel on specific equipment. The scope of the present disclosure includes commercially available task managers, also known as business process management (BMN) tools, such as Process Street, Orchestly, and Process Bliss. According to one embodiment, the task manager 142 is integrated with the AR knowledge base (ie, with the database system 102 used as the knowledge base of the AR system 101).

[0320] Fig.12A process list 1200 is shown displayed to a user of the AR glasses via enhanced AR glasses according to one embodiment of the present disclosure. In the exemplary but non-limiting embodiment shown, the process list 1200 includes three processes 1202A, 1202B, and 1202C. Each process 1202 includes a user assignment 1204, a process line and / or process unit assignment 1206, a description of the process 1208, a process target time 1210, a process real time 1212 (including a process acceptance time (e.g., see 1214A, 1214C) and a process completion time (e.g., see 1216A, 1216C)), a process status 1218, and a user-selectable task link 1220. The task link 1220 is a "clickable" link, as discussed above in the previous embodiment. Upon selection of the “clickable” link by the user, the AR glasses send a signal to the program logic instructing the program logic to retrieve additional data corresponding to the selected process (e.g., process 1204B is the selected process when the user selects process 1240B by clicking on user-selectable task link 1202B associated with process 1204B) from the database system. In one embodiment, the additional data is defined as the first (or next) task to be performed in the corresponding process, and the AR glasses then display the additional data, i.e., the first (or next) task, to the user via the AR glasses, as further discussed more fully below.

[0321] In one embodiment, user assignment 1204 is the name of the user, or if the process has not been assigned to a user, the user assignment is "open", as shown in user assignments 1204B and 1204C. Process line and / or process unit assignment 1206 can describe a specific process line or a specific process unit in the plant, where a process line includes two or more equipment that provides a specific process or multiple processes (e.g., multiple sequential processes), and a process unit is a single equipment for providing a single (or multiple) process. Description of the process 1208 describes the process, and process target time 1210 describes the time when the first task in the task list will start, or in an alternative embodiment describes the latest time when the first task in the task list will start. Process acceptance time 1214 is the time when the first task actually started, process completion time 1216 is the time when the last task in the task list actually completed, and process status 1218 is the state of the process. For example, the state of the process includes "in progress", indicating that the first task has been started (i.e., initiated) but the last task has not yet been completed, and "open", indicating that the first task has not yet been initiated. In another embodiment, the status includes "closed", indicating that the last task of the corresponding process has been completed.

[0322] In one embodiment, as shown, the user-selectable task link 1220 is "Show Details", and when the link 1220 is selected by the user, the AR glasses then display additional data (e.g., according to one embodiment, the first (or next) task is displayed to the user via the AR glasses). The user-selectable link 1220 is also referred to as a selectable virtual object, which may have an associated corresponding secondary virtual object, as discussed in previous embodiments. Upon receiving a selection of the selectable virtual object 1220, the task manager 142 accesses the database system 102 to obtain the corresponding secondary virtual object associated with the selected virtual object 1220. The corresponding secondary virtual object is defined as a task (e.g., a first task or a next task) that is associated with a process (assigned to the user) and corresponds to the selected virtual object 1220 (i.e., "Show Task").

[0323] As previously discussed above in conjunction with other embodiments, and thereby incorporated into the present embodiment, the user can select link 1220 via hands-free voice selection (e.g., by speaking the color of link 1220 (for those embodiments that include colored selectable virtual objects) or by speaking "show details"), via hands-free gaze selection (i.e., the AR glasses and / or the positioning system include one or more directional sensors that are configured to detect the line of sight of the AR glasses or the eye orientation of the user wearing the AR glasses), or via the user simply pointing a hand and / or finger at the selectable link 1220 such that a camera of the AR glasses detects the position of the hand and / or finger.

[0324] Fig.13 The first task 1300 displayed as viewed by the user via the AR glasses according to one embodiment of the present disclosure is shown. In the case where the status of a process is "in progress", indicating that at least the first task of the process has been previously completed, the present embodiment also covers the case where the first task displayed is the "next task". The task manager 142 is configured to: update the status of the process stored in the database system 102 based on the last task completed, and store data indicating the next task to be provided to the display of the AR glasses when the user selects the user-selectable link 1220. In one embodiment, the task manager dynamically updates parameters associated with each process. Parameters include such as Fig.12 One or more of 1204, 1206, 1208, 1210, 1212 (ie, 1214, 1216), 1218, and 1220 as shown.

[0325] As shown, when the user selects user-selectable task link 1220, a first task 1300 (also referred to as additional data) is displayed. The displayed first task 1300 includes a task details box 1302, which includes, but is not necessarily limited to, one or more of the following: a user-selectable confirm task completion link 1304, which, when selected by the user, confirms the completion of the first task; a user-selectable upload document link 1306, which, when selected, enables the user to upload photos and / or videos of the equipment on which the task was performed that the user can take; and / or text describing or annotating the performed task; short text 1308, which briefly describes the task to be performed; detailed text 1310, which describes the task to be performed in more detail; photos 1312, which includes one or more photos of one or more equipment on which the first task is to be performed; and a user-selectable video link 1314, which, when selected by the user, plays a video describing the task to be performed.

[0326] By way of illustrative example, the short text 1308 briefly describes the task to be performed (e.g., "Start dosing line 1"), which is combined with Figure 5 The detailed text 1310 describes in more detail also in conjunction with Figure 5 (Delivery unit (ie, labeled delivery and heating equipment)) tasks to be performed (eg, "main switch on, check set points of dosing and delivery unit, press start"). However, despite Figure 5 1 shows the view of the user wearing the AR glasses when the user looks at the real equipment in the factory, including the display of virtual objects A1 to A7 covering the view of the real equipment, but the present embodiment displays the displayed task 1300 and the displayed process list 1200 ( Fig.12 ) is provided to the user regardless of the user's location (i.e., positioning) relative to the factory or relative to the proximity of equipment in the factory.

[0327] The task details box 1302 may also include one or more of the user-selectable pause links 1316, which, when selected by the user, instruct the program logic to store the current state of the process (including the last task completed) in the database system and pause and / or end the user's current session (e.g., the task details box 1302 is no longer displayed or the AR session with respect to the current user ends). In one embodiment, upon selection of the pause link, the program logic is configured to remove the task details box 1302 from display in the AR glasses, and upon removal, display the task details box 1302 in the AR glasses as shown in FIG. Fig.12The process listing 1200 is shown. As described in more detail below, the task details box 1302 may also include a user-selectable navigation route link 1318 and / or a user-selectable robot command link 1320.

[0328] According to one embodiment, upon selection of the user selectable confirmation task link 1304 by the user, the executable logic is further configured to receive the next task from the database system (e.g., from the task manager 142) for display to the user. The display of the next task includes a task details box having the same Fig.13 The same format as the task details box 1302. For example, Fig.14 FIG. 14 shows a next displayed task 1400 including a task details box 1402 displayed to the user of the AR glasses according to one embodiment of the present disclosure. Fig.14 As shown, the short text 1408 associated with the "next" task briefly describes the Figure 5 The next task to be performed by the packaging equipment shown (i.e., "line 1 starts packaging unit 1235"), and the detailed text 1410 describes in more detail the next task to be performed also in conjunction with the packaging equipment (i.e., "main switch turned on, check the set point of the packaging unit, press start"), such as Figure 5 shown.

[0329] In addition, the executable program logic is further configured to: upon receiving a signal indicating a selection by the user of a task completion link confirming completion of the next task by the user, iteratively receive each subsequent next task upon receiving a signal indicating completion of a previously sent task, until the one or all of the tasks corresponding to the process selected by the user are completed, or until the user selects a user-selectable pause link.

[0330] In another embodiment, and when all of the one or more tasks corresponding to the process selected by the user are completed, or when the user selects the user-selectable pause link (which indicates a pause in further task completion or, in another embodiment, indicates the end of the current AR session), the task manager 142 is further configured to: update the state of the process having the task selected by the user, and store the updated state of the process in the database system. As discussed above, the state can be one of the following states: process "completed" or process "in progress".

[0331] In one embodiment, the AR glasses are further configured to: take a photo or video when the user selects the user-selectable upload document link 1306, wherein the initiation of the taking of the photo or video is based on the action of the user wearing the AR glasses. For example, in one embodiment, the user may issue a voice command for taking a photo or video. The executable program logic is configured to receive the voice command, such as "take a photo" or "take a video". The program logic then sends a photo / video initiation signal to the AR glasses, thereby enabling the taking of photos and / or videos of one or more equipment or the surrounding environment of the equipment by the user positioning the AR glasses or positioning the user's eyes in any of the previously described embodiments, and according to the embodiments discussed previously, it captures the photo or initiates the capture of the video after a predefined time interval after the positioning.

[0332] In another embodiment, the database system 102 is further configured to store spatial coordinates associated with the corresponding equipment in the plurality of equipment, and the augmented reality system further includes a positioning system 108 ( Figure 1A ). The positioning system 108 is configured to sense the spatial position of the user in the factory, as discussed in detail above in conjunction with other embodiments. The executable program logic is further configured to: receive the latest spatial position of the user wearing the augmented reality glasses from the positioning system 108 and receive spatial coordinates of one or more equipment associated with one or more tasks corresponding to the process selected by the user (i.e., process tasks) from the database system 102.

[0333] In addition, the program logic is further configured to: determine a navigation route from the user's latest spatial position to one or more equipment associated with the currently displayed task (e.g., 1300 or 1400), and then send the navigation route to the AR glasses for display to the user.

[0334] For example, if the user wishes to be navigated to one or more equipment associated with the current task to be performed, the user selects the navigation route link 1318. In response, a navigation command signal is generated and sent from the AR glasses to the program logic, and the program logic combines access to the database system to obtain the spatial coordinates of the one or more equipment and the latest spatial position of the user (which can alternatively be obtained directly from the positioning system 108) to determine the route (or one or more different routes) from the spatial position of the user to the one or more equipment. The user can then traverse the route displayed to the user via the AR glasses to reach the one or more equipment. The user can then perform the task as displayed simultaneously (e.g., the displayed task 1300 or 1400 displayed simultaneously with the navigation route in the AR glasses). In another embodiment, the program logic is configured to: only display the navigation route when the user initially moves toward the one or more equipment, receive updates to the user's spatial position when the user traverses the navigation route, and only display the task (e.g., 1300 or 1400) when the user reaches the one or more equipment (e.g., within 1m, 3m, or 5m of the one or more equipment).

[0335] In yet another embodiment, the automation system further includes a distributed control system 132 ( Figure 1A ), the distributed control system is communicatively coupled to the database system 102 and is also communicatively coupled to one or more robotic units in the factory (e.g., Figure 1A 124), also referred to as robot 124. In this embodiment, the distributed control system 132 is configured to receive the latest spatial position of the robot unit in the factory. For example, some robot units may not be stationary, but may be configured to move around the factory. The trajectories of these mobile robot units 124 are determined via pre-programmed instructions stored in a robot unit memory (not shown) and executed by a robot unit processor or microcontroller (not shown), or via commands received from control software (i.e., robot control software) executed by a processor 138 of the distributed control system 132. The executable program logic is further configured to: further determine one or more alternative navigation routes starting from the latest spatial position of the user to the one or more equipment based on the latest spatial position of the one or more robot units in the factory. Therefore, when, for example, the fastest or shortest route is compromised by a potential collision with a user or an obstruction by a user, the system can advantageously provide an alternative route based on the predicted position of the robot unit when the user traverses the route.

[0336] In another embodiment, or in addition to the above embodiments, the distributed control system 132 is also communicatively coupled to the sensors 112, 114, 116 ( Figure 1A ), such as motion sensors, light sensors, heat sensors, chemical sensors, and / or radiation sensors located on infrastructure and / or equipment in the laboratory, including sensors associated with and / or located on the robotic unit 124. In this embodiment, the executable control software 136 is configured to: detect hazardous plant conditions and / or dangerous plant conditions due to fire, radiation leaks, chemical leaks, etc. via sensor signals, and send signals representing the coordinates of areas of the plant to be avoided due to these perceived hazards and / or dangers (i.e., the spatial location or spatial boundaries of the area). The program logic 106 is further configured to: determine one or more alternative navigation routes from the latest spatial location of the user to the one or more equipment based on the spatial location of the area to be avoided, with or without the latest spatial location of the one or more robotic units in the plant. Therefore, when, for example, the fastest or shortest route is compromised by the user's potential intersection with an area of ​​the plant that is considered hazardous and / or dangerous when the user traverses the route, the system can advantageously provide an alternative route.

[0337] In another embodiment, the displayed tasks (e.g., 1300 and 1400) include a user-selectable robotic command link 1320 for directing a robotic unit to perform a first task. In this embodiment, the executable program logic is further configured to: determine a robotic navigation route for directing the robotic unit to the one or more equipment starting from the robotic unit's most recent spatial position, and send the robotic navigation route and instructions for the robot to perform the task to the robotic unit.

[0338] For example, if the user decides that the robot unit performs the current task displayed to the user, the user selects the robot command link 1320 to select the robot unit and direct the robot unit to the corresponding equipment for performing the task. In response to the selection of the robot command link 1320, a robot command signal is generated and sent from the AR glasses to the program logic, and the program logic determines a route from the latest spatial position of at least one robot unit to the one or more equipment in combination with accessing the database system to obtain the spatial coordinates of the one or more equipment corresponding to the task to be performed and the latest spatial position of the one or more robot units.

[0339] In one embodiment, a route is determined starting from the latest spatial position of a robotic unit that is both closest in position to the one or more equipment and operable to perform the task. In the meaning of this embodiment, "operable" to perform the task means that the robotic unit is programmable to perform the task and is available to perform the task (i.e., not currently performing any other task in the factory). Information corresponding to the corresponding robotic unit (such as the corresponding programmable task) can be stored in the database system and can be updated by the distributed control system 132 based on, for example, an update to the operating system of the robotic unit or the executable software of the robotic unit, and a copy of the update can be stored in the memory 134 of the control system 132.

[0340] The program logic then sends the robot navigation route and instructions for the robot to perform the task to the robot unit selected to perform the task. In one embodiment, the instructions can be a signal identifying a subroutine or application of executable robotic unit software (not shown) that is configured to instruct the unit to perform the task and enables initiation of execution of the subroutine or application when the robotic unit reaches the one or more equipment.

[0341] In another embodiment, the robotic unit (selected to traverse the route and perform the task) is configured to send a task completion signal to the database system when the task is completed. The executable program logic is further configured to receive the next task to the AR glasses for display to the user when the task completion signal is received by the database system. The user can then decide to perform the next task, or as previously described, enable the robotic unit 124 to perform the task.

[0342] In another embodiment, the database system 102 and / or the distributed control system 132 include a user database 140. The user database is configured to store user information associated with the production process in the factory. In one embodiment, the user information includes, but is not limited to, one or more of a user ID, a user role, and / or user permissions. In another embodiment, the executable program logic is configured to: receive user credentials from the user, authenticate the user, and send the process list to the AR glasses for display to the user of the AR glasses if and only if the user is authenticated. The user can be authenticated by comparing the received user credentials with the user information stored in the user database based on the program logic. As discussed above in conjunction with other embodiments, the user credentials can be input into the automation system by the user using, for example, a keyboard or voice command via a wireless connection or a wired connection, or the user credentials can be received by the automation system via a scanning device. The scope of this embodiment covers encryption of the user information and user credentials, including the use of keys and passwords, to provide digital security in the user authentication process.

[0343] In another embodiment, the database system 102 is a graph database system, a spatial database system, and / or a streaming database system. The graph database system includes a graph database having a plurality of interconnected nodes (e.g., see Figure 3 300; Fig.13 1302; and / or Figures 1400, 1402, where each task instruction box, also referred to as a task or task instruction, may be formed as a component of a node as shown. Each node also represents one or more objects of the industrial plant and includes spatial information associated with the one or more objects. In one embodiment, each of the one or more objects includes one of the following: equipment, sensors, actuators, and raw materials in the plurality of equipment. In one embodiment, the equipment includes one or more robot units (e.g., robot unit 124).

[0344] In yet another embodiment, one or more of the nodes include at least one corresponding container or are stored in association with the at least one corresponding container. Each container is an isolated runtime environment instance that includes software for monitoring and / or operating at least one of the one or more objects represented by the corresponding node, and the software in each container includes a location reporting module that is configured to monitor the location of at least one of the one or more objects of the corresponding node and update the spatial information associated with the at least one or more objects of the corresponding node in the database system 102.

[0345] Fig.15A flow chart of a method 1500 for controlling an automation system according to an embodiment of the present disclosure is shown, wherein the automation system is used to control an automated production process in an industrial factory. For example, the method may be based on Figure 1A The automation system shown is performed. An industrial plant includes multiple equipment 122, 124 for performing a production process. The multiple equipment may be spatially distributed in the industrial plant. The automation system includes: an augmented reality system 101, the augmented reality system including: a database system 102 configured to store data associated with corresponding equipment in the multiple equipment; augmented reality glasses 104 configured to display the data; and executable program logic 106 coupled to the augmented reality glasses. The database system includes a task manager 142 configured to manage processes relative to one or more equipment. The data includes a list of processes managed by the task manager. Each process includes one or more tasks to be performed relative to one or more equipment.

[0346] In optional step 1502, the executable program logic 106 sends a request for a list of processes to the database system 102. In one embodiment, the task manager 142 of the database system 102 receives the request. In another embodiment, the task manager continuously or periodically receives one or more of factory sensor data, equipment control data, and data from the database system corresponding to the status of a current or upcoming production process and / or a process updated based on data received from the robotic unit 124 and / or one or more users of one or more AR glasses, and continuously or periodically updates the list of processes.

[0347] In step 1504, the program logic receives the list of processes from the database system (e.g., from the task manager of the database system) for display to the user of the AR glasses. At least one of the processes is assigned to the user of the AR glasses.

[0348] In step 1506 , the program logic receives a selection made by the user of the AR glasses of a displayed process of the at least one display process assigned to the user.

[0349] In step 1508, the program logic sends the selected process to the database system (eg, to the task manager of the database system). In one embodiment, the program logic sends a signal indicating the selection of the displayed process to the database system.

[0350] In step 1510, the program logic receives additional data corresponding to the selected process from the database system. In one embodiment, the task manager 142 of the database system 102 processes the selected process and accesses the additional data (corresponding to the selected process) stored in the database system (e.g., stored in the database 110), or directly accesses the additional data (corresponding to the selected process) based on data stored in the queue or memory of the task manager, and sends the additional data to the executable program logic 106 of the AR glasses for display to the user of the AR glasses.

[0351] Various aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each frame of the flowchart illustration and / or block diagram and the combination of frames in the flowchart illustration and / or block diagram can be implemented by computer-readable program instructions.

[0352] Embodiments of the present disclosure may be systems, methods, products, and / or computer program products. A computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions for causing a processor to perform various aspects of the present invention.

[0353] A computer-readable storage medium may be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punched card or a raised structure in a groove on which instructions are recorded) and any suitable combination of the above. As used herein, a computer-readable storage medium will not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

Claims

1. An automation system (100) for controlling an automated production process of an industrial plant, the industrial plant comprising a plurality of equipment (122, 124) for executing the production process, the plurality of equipment being spatially distributed in the industrial plant, the automation system include: - an augmented reality system (101), the augmented reality system comprising: o a database system (102) configured to store data associated with respective equipment of the plurality of equipment, the database system comprising a task manager (142) configured to manage processes with respect to one or more equipment, the data comprising a list of processes managed by the task manager, each process comprising one or more tasks to be performed with respect to the one or more equipment; o augmented reality (AR) glasses (104), the augmented reality (AR) glasses being configured to display the data; and o executable program logic (106), the executable program logic coupled to the augmented reality glasses and the database system, the executable program logic configured to: receiving from the database system the list of processes for display to a user of the AR glasses, at least one of the processes being assigned to the user of the AR glasses, receiving a selection by the user of the AR glasses of a process to be assigned to a display of the user, Sending the selected process to the database system, and • Receiving from the database system additional data corresponding to the selected process for display to the user of the AR glasses.

2. The automation system of claim 1 , wherein each process in the displayed process list includes one or more of: user assignment, process line and / or process unit assignment, a description of the process, a process target time, a process acceptance time, a process completion time, a process status, and a user-selectable task link, wherein the task link enables a task of the one or more tasks to be displayed in the AR glasses.

3. The automation system of claim 2 , wherein the executable program logic is configured to: receive the selection of the displayed process assigned to the user by receiving a selection of the task link, and send the selected process to the database system by sending the selected task link to the database system, wherein the additional data includes a first task of the one or more tasks associated with the selected task link.

4. An automated system according to claim 3, wherein the first task displayed to the user includes one or more of the following: a description of the first task, a detailed description of the first task, a user-selectable link to a video of the first task, one or more photos of the one or more equipment on which the first task is to be performed, a user-selectable link for uploading photos, and a user-selectable task completion link confirming completion of the first task by the user.

5. The automated system of claim 4, wherein upon selection by the user of the user-selectable task completion link, the executable logic is further configured to: receiving from the database system a next task for display to the user of the AR glasses, and Upon receiving a signal indicating selection by the user of the task completion link confirming completion of the next task by the user, iteratively receiving each subsequent next task upon receiving a signal indicating completion of a previously sent task, until all of the one or tasks corresponding to the process selected by the user are completed, or until the user selects a paused user-selectable link indicating further task completion of the remaining tasks of the one or more tasks in the process selected by the user.

6. The automation system of claim 5, wherein when all of the one or tasks corresponding to the process selected by the user are completed, or when the user selects a user-selectable link indicating a pause for further task completion, the task manager is further configured to: updating a status of a process selected by the user, wherein the status is one of: process completed or process in progress; and The updated status of the process is stored in the database system.

7. The automated system of any one of claims 4 to 6, wherein upon selection by the user of the user-selectable task link for uploading a photo, the AR glasses are further configured to: Taking a photo or a video, wherein initiating the taking of the photo or the video is based on an action of the user wearing the AR glasses.

8. The automated system of claim 7, wherein the actions of the user comprise hands-free actions.

9. The automated system of claim 8, wherein the hands-free action of the user comprises one of: an orientation of the AR glasses, an orientation of the user's eyes, and a voice command.

10. The automation system according to any one of claims 4 to 9, wherein the database system (102) is further configured to store spatial coordinates associated with the respective equipment among the plurality of equipment, wherein the augmented reality system further comprises a positioning system (108) configured to sense a spatial position of the user in the factory, and wherein the executable program logic is further configured to: receiving from the positioning system (108) an updated spatial position of the user wearing the augmented reality glasses and receiving from the database system spatial coordinates corresponding to the one or more equipment associated with the one or more tasks corresponding to the process selected by the user; determining a navigation route from the latest spatial location of the user to the one or more devices; as well as The navigation route is sent to the AR glasses for display to the user.

11. The automation system according to claim 10, further comprising: include: a distributed control system (132) coupled to the database system and to one or more robotic units in the factory, the distributed control system being configured to receive an updated spatial position of the one or more robotic units in the factory, wherein The executable program logic is further configured to determine one or more alternative navigation routes from the latest spatial position of the user to the one or more equipment based on the latest spatial position of the one or more robotic units in the factory.

12. The automation system of claim 11, wherein the displayed first task further comprises a user-selectable link for directing the robotic unit to perform the first task, and wherein the executable program logic is further configured to: determining a robotic navigation route for guiding the robotic unit to the one or more equipment starting from the latest spatial position of the robotic unit; and The robotic navigation route and instructions for performing the first task are sent to the robotic unit.

13. The automation system of claim 12, wherein the robotic unit is configured to send a task completion signal to the database system when the first task is completed, and wherein the executable program logic is further configured to receive a next task to the AR glasses for display to the user when the task completion signal is received by the database system.

14. An automation system according to any one of the preceding claims, wherein the database system further comprises a user database (140), the user database being configured to store user information associated with the production process in the factory, wherein the user information comprises a user ID, a user role and / or user permissions.

15. The automation system of claim 14, wherein the executable program logic is configured to: receiving user credentials from the user; authenticating the user; and If and only if the user is authenticated, the process list is sent to the AR glasses for display to the user of the AR glasses.

16. An automation system according to any one of the preceding claims, wherein the database system is a graph database system, a spatial database system and / or a streaming database system, wherein the graph database system includes a graph database having a plurality of interconnected nodes (300), wherein each node represents one or more objects of the industrial plant and includes spatial information associated with the one or more objects, and wherein each of the one or more objects includes one of the following: equipment, sensors, actuators and raw materials among the plurality of equipment, wherein the equipment includes one or more robot units.

17. An automation system according to claim 16, wherein one or more of the nodes include at least one corresponding container or are stored in association with the at least one corresponding container, each container is an isolated runtime environment instance, the runtime environment instance includes software for monitoring and / or operating at least one of the one or more objects represented by the node, and wherein the software in each container includes a position reporting module, the position reporting module is configured to monitor the position of at least one of the one or more objects of the corresponding node and update the spatial information associated with the at least object of the corresponding node in the database system.

18. The automation system according to any one of claims 1 to 17, wherein the task manager is configured to assign processes in the process list to one or more users and to dynamically update parameters associated with each process.

19. A method for controlling an automated production process of an industrial plant, the industrial plant comprising a plurality of equipment (122, 124) for executing the production process, the plurality of equipment being spatially distributed in the industrial plant, the method include: -Providing an automation system, the automation system comprising an augmented reality system, the augmented reality system comprising: o a database system (102) configured to store data associated with respective equipment of the plurality of equipment, the database system comprising a task manager configured to manage processes with respect to one or more equipment, the data comprising a list of processes managed by the task manager, each process comprising one or more tasks to be performed with respect to the one or more equipment; o augmented reality glasses (104) configured to display the data; and o executable program logic (106) coupled to the augmented reality glasses and the database system; - receiving from the database system the list of processes for display to a user of the AR glasses, at least one of the processes being assigned to the user of the AR glasses; - receiving a selection by the user of the AR glasses of a process assigned to a display of the user; - sending the selected process to said database system; and Additional data corresponding to the selected process for display to the user of the AR glasses is received from the database system.

Images