AUGMENTED REALITY SYSTEM FOR TEACHING A USER OF AN APPLICATION

Description

[0001] The invention relates to an augmented reality system for teaching a user of an application as well as to a corresponding method and a digital terminal for such an augmented reality system.

[0002] Today's applications, especially industrial applications, increasingly require human-machine interaction. In particular, collaboration between humans and, for example, industrial robots is becoming increasingly spatially closer and more interactive.

[0003] Such human-machine interactions pose an increased risk. For example, such interactions can lead to accidents that can cause human injuries. Furthermore, productivity deficits arise if the interaction between humans and machines is not optimally coordinated. Consequently, there is a need to further improve the safety of human-machine interactions and productivity.

[0004] EP 3 546 136 A1 describes an augmented reality system for overlapping an augmented reality with a real environment, in which a current image of an environment and a virtual image are displayed in relation to each other via a display unit. DE 20 2018 212 944 A1 discloses a method for supporting human-robot collaboration using data glasses that visualize location-related information, with acoustic signals being output, among other things.

[0005] An object underlying the invention is to provide a system and a method for instructing a user of an application as well as a corresponding digital terminal for instructing a user of an application, which increase the safety and / or productivity in human-machine interactions.

[0006] This object is achieved by the subject matter of the independent claims. Further advantageous embodiments can be found in the subclaims.

[0007] The invention relates to an augmented reality system (AR system) for teaching a user of an application, comprising: a detection unit that detects a real environment and provides it as environmental data, wherein at least one work unit is arranged in the environment; a communication unit that receives process data from the work unit; a control unit that processes the process data and environmental data, wherein a virtual image can be created based on the process data; a display unit that displays the real environment and the virtual image in relation to one another to a user; and an audio output unit that outputs audio signals based on the process data for instructing a user, wherein the audio signals are preferably matched to the virtual image, in particular contextually and / or causally, wherein instruction comprises one or a plurality of the following components: teaching a user, training a user, instructing a user, supporting a user, instructing a user,Providing instructions regarding the use of the work unit, providing hazard and / or warning information and providing general information, wherein the process data comprises data for identifying a process step of the work unit and the audio output unit outputs an audio signal corresponding to an identified process step and / or the display unit displays a virtual image corresponding to an identified process step, wherein the AR system comprises a memory in which a plurality of audio signal data and / or augmentation data are stored, wherein the audio signal data and augmentation data are assignable to the process steps and / or phases of a work life cycle and are categorized accordingly.

[0008] In other words, a real environment captured by the capture unit is overlaid with an image created using process data from the work unit and displayed on a display unit. The capture unit can, for example, be a camera that records image and / or video data and / or 3D or depth data of the real environment as environmental data. This image and / or video data is displayed as image and / or video recordings on the display unit after being sent to the communication unit and processed by the control unit. To improve instruction of a user, the image and / or video recordings of the real environment are displayed on the display unit with a virtual image, which can include a plurality of virtual objects, in relation to one another, i.e.The AR system processes the environmental data of the real environment and the process data of the work unit to adapt the virtual image to the real environment. In particular, the virtual image is integrated into the real environment in such a way that it appears as if the virtual image is actually located in the real environment. The audio output unit further outputs audio signals based on the process data of the work unit to instruct the user. Preferably, the audio signals are tailored to the virtual image, in particular contextually and / or causally, i.e., the audio signals support the content conveyed by the virtual image.

[0009] Training comprises one or more of the following components: teaching a user, training a user, instructing a user, assisting a user, instructing a user, providing information regarding the use of the work unit, providing hazard and / or warning information, and providing general information. The training can be provided in the form of spoken text, in particular comprising several complete spoken sentences, audio signals, i.e., individual tones, and / or similar. For example, operating instructions for operating the work unit can be provided to the user as an audio output.

[0010] The environmental data can also include data that is provided separately from the data from the recording unit, for example CAD data, e.g. in the form of a virtual surface model of the work unit and / or the environment. In this way, the environmental data can then be used to determine that the work unit is approaching a wall or another object in the environment. For example, crushing hazards can be automatically detected by the AR system if, for example, a tool or another part of the work unit falls below a predetermined distance from an object in the environment. In this way, it can be made possible, for example, to take into account (automatic) subsequent changes to the movements of the work unit. In particular, the training can take into account that the work unit can be enabled to make independent decisions and, for example, carry out modified movements.

[0011] Audio signals can play a key role in increasing safety. This allows the user to perceive the audio signals at any time, even if they are focused on the work unit and cannot perceive graphical information at that moment, thus receiving information that increases their safety.

[0012] In particular, the audio signals can include warnings, rules of conduct and instructions, operating aids or instructions, and / or other audio signals useful for instruction. The virtual image can include, for example, a virtual image of the work unit or a virtual image of the movement of the work unit, geometric shapes such as circles, rectangles, triangles, and / or other geometric shapes, colored markings or highlights, and / or other images useful for instruction, including text-based representations, e.g., of process parameters.

[0013] Further embodiments of the invention can be found in the description, the subclaims and the drawings.

[0014] According to the invention, the process data comprises data for identifying a process step of the work unit, and the audio output unit outputs an audio signal corresponding to an identified process step and / or the display unit displays a virtual image corresponding to an identified process step. In particular, the audio signal and the virtual image are coordinated with one another, i.e., the content of the virtual image is supported by the audio signal. For example, a process step of the application can be represented by the virtual image via the display unit, while at the same time the audio output unit outputs relevant instructions, information, and / or a description of the process step via the audio signal.The user is thus taught through various sensory perceptions, which improves the quality of the instruction and the user's reception of information, and minimizes misinterpretation of the visual information.

[0015] According to the invention, the AR system comprises a memory in which a plurality of audio signal data and / or augmentation data is stored, wherein the audio signal data and augmentation data can be assigned to the process steps and / or phases of a work life cycle (e.g. commissioning or maintenance).

[0016] The audio signal data and / or augmentation data are further categorized, i.e., audio signal and / or augmentation data associated with a specific process step and / or phase of a work life cycle are grouped in the memory. In particular, the audio signal and / or augmentation data can be stored as audio signal and / or augmentation files. Categorizing or grouping the audio signal and / or augmentation data according to process step and / or work life cycle helps to consider different risks during the interaction between user and application. The audio signal and / or augmentation data can be included in the delivery of an application as audio signal and / or augmentation files or can be downloadable via a supplied QR code or via a serial number on a sensor.Alternatively, the audio signal and / or augmentation data can also be stored in a cloud. In this case, the digital device accesses the corresponding audio signal and / or augmentation data via the cloud. This allows the content to be easily updated. Different languages ​​could be made available to specific users on demand.

[0017] According to a further embodiment, the audio output unit is designed to play the audio data and / or the display unit is designed to play the augmentation data before a future process step and / or at the beginning of a process step.

[0018] In particular, the playback of audio and / or augmented data indicates the start of a new process step, a change in circumstances or a change in state, and / or a transition from one process step to the next. The user is thus notified of a change in the current state both by the audio signal and by the image on the display unit. It is sufficient for the user to perceive one of the signals, thus further increasing user safety.

[0019] In particular, the AR system can be implemented as a publisher-subscriber system. For example, the work unit can publish the process step at the beginning of a process step or during the execution of a process step, whereby the digital device subscribes to a corresponding status change and plays corresponding audio and augmentation data. The audio signal and the virtual representation can thus be optimally coordinated with the operation of the work unit.

[0020] According to a further embodiment, the phases of a working life cycle include commissioning, normal operation, maintenance, modification, and / or dismantling of the application. In particular, the phase of the working life cycle can be determined based on the process data. Furthermore, an application can also include additional working life cycle phases beyond those mentioned here.

[0021] According to a further embodiment, the AR system and the work unit are designed to communicate bidirectionally, wherein the AR system is designed to control and / or influence the work unit.

[0022] In other words, data is sent from the AR system to the work unit and vice versa. In contrast to unidirectional communication, where, for example, only the work unit sends data to the AR system, bidirectional communication allows inputs in the AR system to lead to a reaction or action in the work unit, i.e., the operation of the work unit can be actively influenced via the AR system. For example, the execution of a process step can be selected and / or repeated and / or reset using the digital terminal. This makes it possible to "stop," "fast-forward," and / or "rewind" the execution of the process step, i.e., the user can freely select an execution time, for example, via a timeline that corresponds to an execution period of the process step.Additionally or alternatively, N repetitions of the same process step can occur and the AR system can automatically proceed with the execution of the next process step after the N repetitions have been executed.

[0023] According to a further embodiment, the display unit is designed to display text messages. In particular, the text messages can be tailored to the audio signals and the virtual image or generated from them, for example by automated audio-to-text converters. In addition to such transcribing, translation is also possible, i.e., the audio data can be presented in another language as text, similar to a subtitle. Furthermore, the text messages can be stored as text data in the memory and assigned to corresponding audio and augmentation data. In addition to the audio signal and the virtual image, the text messages can support the user during instruction. The text messages can be displayed on the display unit and contain a subtitle, action and / or behavioral instructions, or-guidelines, a name for a process step, warnings, and / or other information relevant to a process step. Audio and / or text data can also be available and output in different languages.

[0024] According to a further embodiment, the AR system is designed to check a user's training level by means of a test, in particular a multiple-choice test, and to enter and / or update the user's training level in a user database based on a test result.

[0025] For example, the test can be administered automatically after completing a training session. The test result can be determined based on the number of correctly answered questions, the number of incorrectly answered questions, and / or the ratio between correctly answered and incorrectly answered questions. The determined test result can be assigned to a training level, so that the training level is determined based on the test result. Preferably, the user's training level can comprise two or more statuses. For example, the user's training level can be "passed" once a certain number of questions are answered correctly, and "failed" if the specified number of correctly answered questions is not reached. The training level can also comprise a qualification level.For example, the user's qualification level can include the statuses "Novice", "Experienced User" and "Expert", where, for example, in a questionnaire of ten questions, correctly answering zero to four questions can lead to the status "Novice", correctly answering five to seven questions can lead to the status "Experienced User" and correctly answering eight to ten questions can lead to the status "Expert".

[0026] It is also possible for the training level and the skill level to be separate metrics. For example, the training level—as described above—can indicate whether the user has successfully completed a training session, while the skill level is determined in advance, independently of the training level.

[0027] Furthermore, answers to test questions can be entered via a touch display. For example, in such a case, the display unit can be designed as a touch display. Additionally or alternatively, the user's answers to test questions can be entered into the AR system via a microphone, and the spoken language or spoken answers can be recognized using speech recognition software.

[0028] The display unit can, for example, comprise a screen on which the environmental data (e.g., an image or video captured by the acquisition unit) is displayed, with the virtual image superimposed on it. The display unit can also comprise AR glasses. With AR glasses as the display unit, the real environment can be displayed to the user through an at least partially translucent part of the AR glasses, with the virtual image projected so that it is visible to the user.

[0029] Upon successful completion of the test, a certificate can be issued, which is stored in the user database. The certificate can expire after a specified period of time, for example, six months. The expiration date or validity period can be stored in the user database. After the certificate expires, the user can be prompted to refresh a training level via the AR system or digital device.

[0030] For example, the user database stores data about a user, particularly data that reflects information about a user's qualification level with regard to a training session. Other data in the user database includes, for example, an employee number, a first name, a last name, a job title, a training status with regard to an application, a training certificate, a date of the last training session and / or an expiration date of the certificate corresponding to a training session and / or a validity period of the certificate corresponding to a training session, a qualification level, a user's biometric data, and / or other data.

[0031] According to a further embodiment, the AR system comprises a training level determination unit which is designed to determine an identity of the user based on personal identification data and to determine the training level of the user based on the determined identity by means of the user database.

[0032] For example, personal identification data can be captured via a device number of the digital device, via a person's login to a database system using a password, or via a person's biometric data, such as fingerprint and / or facial recognition. For example, the user's fingerprint and / or face can be captured using a fingerprint sensor and / or a camera and compared with a fingerprint and / or face of at least one user stored in the user database to determine the user's identity.

[0033] According to a further embodiment, the working unit is designed to set an operating mode of the working unit based on the training level of the user.

[0034] For this purpose, the work unit can receive data on the user's training level and design or adapt safety parameters based on the received data. For example, after successfully identifying the user and successfully determining the user's training level, the AR system can send the data on the user's training level to the work unit, which can then, for example, adapt the speed at which the work unit moves to the user's training level. For a user who, for example, has a valid training certificate stored in the user database, the work unit can be operated in normal mode, i.e. without a reduction in speed, whereas for a user for whom no valid training certificate is stored in the user database, the work unit is operated at reduced speed or stops working.By adapting the operating mode of the work unit to the user's level of training, increased safety and productivity can be ensured.

[0035] According to a further embodiment, the AR system is designed to limit or expand a usage area and / or contents of the AR system based on the user's level of instruction.

[0036] For example, usage rights and access to AR system content can be set based on the user's training level and / or qualification level. For example, based on their individual training level and / or qualification level, the user can have usage rights to use predetermined functions of the AR system that are assigned to their training level. The functions of the AR system can include, for example, conducting various training sessions, controlling the robot, and / or accessing various AR system content. Users can be classified into the qualification levels "expert," "experienced user," and "beginner." A user with the qualification level "expert" can use the full functionality of the AR system and, for example, also access content related to commissioning and maintenance of the work unit.For example, a user with the qualification level "experienced user" can also use all functions of the AR system and has access to information about changes in the application; however, no access to content related to commissioning and maintenance of the work unit. A user with the qualification level "beginner", on the other hand, can only use certain predefined functions of the AR system, also receives information about changes in the application, and has access to further training opportunities in the form of instructions, for example, to increase the qualification level and become an "experienced user". In other words, accessibility to functionalities of the AR system is set based on the qualification level and / or the level of instruction of the user, i.e., the area of ​​use of the AR system is tailored to the experience oradapted to the user's knowledge, thus creating increased security when using an application.

[0037] According to a further embodiment, the real environment and / or the work unit comprises a marker which is used by the AR system as a reference point and / or to recognize the work unit.

[0038] In particular, the marker is used as a reference point for augmented objects and in particular for positioning the augmented objects in three-dimensional space. This enables an improved and realistic representation of the augmented objects on the display unit. Additionally or alternatively, the AR system can use the marker to determine a work unit type. For example, the work unit can comprise a robot, and the marker can be used to determine a robot type, a robot model, and / or other properties of the robot. The marker can, for example, comprise a QR code that is detected by the detection unit of the AR system and clearly defines the type and nature of the work unit. This enables easy identification of the work unit.

[0039] According to a further embodiment, the detection unit is spatially separated from the display unit and the communication unit, i.e. the detection unit is not located in the immediate vicinity of the display unit and the communication unit, and sends the environmental data to the communication unit by means of telecommunications.

[0040] For example, the recording unit can be located in close proximity to the work unit, while the display unit and communication unit are not in close proximity to the work unit. In particular, the recording unit and the display unit or communication unit can be located in different rooms or buildings. For example, the recording unit can communicate with the display unit and communication unit via a LAN, VPN and / or mobile phone connection to exchange data. This enables remote training of the user, i.e., a user who is not in close proximity to the work unit can receive training remotely.

[0041] According to a further embodiment, a monitoring unit is configured to monitor a monitoring space and to send security data to the communication unit, wherein the security data is processed by the control unit, and the audio output unit is configured to output an audio signal when the processing of the security data indicates that the user enters the monitoring space of the monitoring unit.

[0042] For example, the audio signal comprises a warning signal which alerts the user that he has entered a safety-critical area, i.e. the monitoring room. In particular, the work unit can additionally change the operating mode, i.e. for example reduce or increase the speed, or discontinue work operation, i.e. stop, when the user enters the monitoring room. Additionally or alternatively, the audio output unit can be designed to output the audio signal without the occurrence of a causal event which was triggered, for example, by the user and / or the work unit. For example, the audio signal can comprise an auditory operating manual, i.e. the audio signal comprises a sound track which is played independently of external events.

[0043] It is also conceivable that the AR system could be used not exclusively to instruct a user, but also, for example, to communicate causal steps that the user of the application should perform. For example, instructions could be communicated to the user via the audio output unit. In such a case, the audio signal would include instructions such as "Task completed, please remove material," "Resupply needed," etc. Thus, the AR system could also be used as an operating aid.

[0044] The invention further relates to a method for teaching a user of an application by means of an augmented reality system, which comprises: a real environment is captured and provided as environmental data, wherein at least one work unit is arranged in the environment; process data is received from the work unit; the process data and environmental data are processed, wherein a virtual image is created based on the process data; the real environment and the virtual image are displayed to a user in relation to one another; based on the process data, audio signals are output to instruct a user, wherein the audio signals are preferably matched to the virtual image, wherein instruction comprises one or a plurality of the following components: teaching a user, training a user, instructing a user, supporting a user, instructing a user, providing information regarding use of the work unit, providing hazard and / or warning information, and providing general information,wherein the process data comprises data for identifying a process step of the work unit and the audio output unit outputs an audio signal corresponding to an identified process step and / or the display unit displays a virtual image corresponding to an identified process step, wherein the AR system comprises a memory in which a plurality of audio signal data and / or augmentation data are stored, wherein the audio signal data and augmentation data are assignable to the process steps and / or phases of a work life cycle and are categorized accordingly.

[0045] The invention also relates to a digital terminal for an augmented reality system for teaching a user of an application, comprising: a communication unit that receives process data from at least one work unit and environmental data from at least one detection unit that detects a real environment, wherein the work unit is arranged in the environment; a control unit that processes the process data and environmental data, wherein a virtual image can be created based on the process data; a display unit that shows a user the real environment and the virtual image in relation to one another; and an audio output unit that outputs audio signals based on the process data for instructing a user, wherein the audio signals are preferably matched to the virtual image, wherein instruction comprises one or a plurality of the following components: teaching a user, training a user, instructing a user, supporting a user, instructing a user, providing information regarding use of the work unit, providing hazard and / or warning information, and providing general information,wherein the process data comprises data for identifying a process step of the work unit and the audio output unit outputs an audio signal corresponding to an identified process step and / or the display unit displays a virtual image corresponding to an identified process step, wherein the AR system comprises a memory in which a plurality of audio signal data and / or augmentation data are stored, wherein the audio signal data and augmentation data are assignable to the process steps and / or phases of a work life cycle and are categorized accordingly.

[0046] The statements regarding the AR system apply accordingly to the method and the digital terminal, particularly with regard to advantages and embodiments. It is understood that all features and embodiments mentioned herein can be combined with one another, unless explicitly stated otherwise.

[0047] The invention is illustrated below purely by way of example with reference to the drawings. In the drawings: Fig. 1 shows a schematic representation of a real environment in which an augmented reality system is used; Fig. 2 shows a schematic detailed representation of a digital terminal of the augmented reality system; Fig. 3 shows a schematic representation of a real environment in which the AR system and a monitoring unit are used; Fig. 4 shows an embodiment of the AR system in which instruction is provided by means of telecommunications; Fig. 5 shows an example of a user database; Fig. 6 shows a phase sequence diagram of a work unit and an associated audio playback by the AR system.

[0048] Fig. 1 shows a schematic representation of a real environment in which an augmented reality system (AR system) 1 is used. In Fig. 1 the user A is located near a work unit 3, which is represented as an industrial robot and connected to a controller 3a. The controller 3a controls the work unit 3 in its processes using the process data stored in the controller 3a. The process data can include the controlled processes of the work unit 3 as well as information about a process step and / or the work life cycle of the work unit 3. The process data particularly include movement data of the work unit 3. CAD data of the work unit 3 can also be provided in the process data and / or augmentation data, so that, for example, a true-to-scale image of the work unit 3 can be displayed. In this case, the work unit 3 can also include a driverless transport system or another type of processing machine.In this context, movement data includes, for example, control data for controlling the robot or for driving the driverless transport system from one location to another.

[0049] The AR system 1 according to the invention used by user A comprises a capture unit 1a, which captures the real environment with the work unit 3. The capture unit 1a provides the captured information as environmental data. The capture unit 1a preferably comprises a camera that captures spatial depth and / or a wide-angle camera, so that the real environment is captured extensively and in detail. Furthermore, three-dimensional spatial data of the environment can be captured using the capture unit 1a. In this context, environmental data is understood to mean data from the camera that is available as image data or can be converted into image data.

[0050] Furthermore, the AR system comprises a communication unit 1b, which receives the process data from the work unit 3, in particular from the controller 3a of the work unit 2. The AR system 1 further comprises a Fig. 1 Control unit (not shown) that processes the environmental data and the process data.

[0051] Here, the control unit creates a virtual image based on the environmental data and the process data, which is displayed on a display unit 1c of the AR system 1. The display unit 1c represents the real environment and the virtual image in relation to one another. In particular, the images recorded by the acquisition unit 1a can be superimposed on the virtual image in such a way that the impression is created that the virtual image represents one or more objects located in the real environment. The virtual objects of the virtual image can be two-dimensional and / or three-dimensional objects.

[0052] Furthermore, the process data of the work unit 3 can include data about programmed and / or possible movements, in particular movement sequences, of the work unit 3. The process data can also include data about a process step and / or a work life cycle of the work unit 3; for example, the process data can provide information about which process step is executed in a phase of a work life cycle.

[0053] The control unit can process the process data of the work unit 3, for example, in real time and thus adapt the virtual image to real-time operation of the work unit 3. The virtual image can, for example, represent a visualized sequence of movements of the work unit 3, display instructions and / or any two-dimensional and / or three-dimensional image. The virtual image can comprise a plurality of different virtual objects. For example, a first virtual object of the virtual image can represent a virtual representation of the work unit 3, while a second virtual object of the virtual image can represent an instruction, such as an arrow, an exclamation mark, a text field or text bubble and / or a geometric shape such as a circle. The display duration of different virtual objects is preferably coordinated with one another.

[0054] In addition, audio signals are played via an audio output unit 1d, such as a loudspeaker, and made available to the user A. The AR system 1 in Fig. 1 comprises a smartphone, wherein the individual units of the AR system 1 are integrated together in the smartphone. For example, the detection unit 1a comprises a smartphone front camera and / or rear camera, the communication unit 1b comprises a transmitter and / or receiver, the control unit a CPU, the display unit 1c a display, and the audio output unit 1d a loudspeaker. However, the AR system 1 can also comprise AR glasses, a helmet display, an electronic communication device, and / or a touchpad.

[0055] The AR System 1 in Fig. 1 simultaneously comprises a digital terminal 5. However, it should be understood that the digital terminal 5 may also comprise fewer or more components than those described here. For example, the detection unit 1a in Fig. 4 Part of the AR system 1, but not part of the digital device 5.

[0056] When user A starts a training session, the real environment in which work unit 3 is located is recorded via the recording unit 1a and displayed on the display unit 1c on the digital terminal 5. If work unit 3 displayed on the display unit 1c goes through the individual process steps or sequences of work unit 3, a virtual image is generated and displayed on the display unit 1c based on the process data sent by work unit 3, which is intended to help or support user A in completing the training. In addition, audio signals are output via the audio output unit 1d, which are also intended to support user A in completing the training.

[0057] The virtual image can, for example, include a virtual image of the work unit 3 or a virtual image of the movement of the work unit 3, geometric shapes such as circles, rectangles, triangles and / or other geometric shapes, colored markings or highlights and / or other images useful for instruction. Furthermore, the audio signals output by the audio output unit 1c can include warnings, rules of conduct and instructions, operating aids or instructions and / or other audio signals useful for instruction. The audio signal can include a female and / or male voice. In addition, instructional videos in sign language can be output in a window on the display unit, which reproduce a corresponding content of the audio signal.

[0058] The displayed virtual image and the output audio signal can be situational, i.e. reacting to a change in a state, and / or situation-independent, i.e. independent of a change in a state. Fig. 3 An exemplary embodiment of a situational AR system is shown.

[0059] It is also possible for multiple users to be instructed simultaneously via different digital devices of an AR system. When instructing multiple users in this way, each user can, for example, conduct an individual instruction via a digital device. This means that the virtual image displayed on one device can differ from the virtual image displayed on another. This allows the instruction process and content to be adapted to each individual user.

[0060] Fig. 2 is a schematic representation of the digital terminal 5, which comprises the display unit 1c, the audio output unit 1d and the communication unit 1b and control unit (not shown). The real environment is shown on the display unit 1c based on the environmental data of the detection unit 1a. The work unit 3 (industrial robot), which is located in a safety area 7, is located in the real environment detected by the detection unit 1a. The safety area 7 defines, for example, an area in which the work unit 3 operates, i.e., outside the safety area 7, a collision between a user A and the work unit 3 is not possible. If a user A undergoes training using the AR system 1, the movements and in particular the movement radius of the work unit 3 are demonstrated to him on the display unit 1c by means of a virtual representation 9 of the work unit 3.In addition to the virtual representation 9 of the work unit 3, the safety area 7 is highlighted in color using another virtual image, for example, by marking the safety area with a striking color. In addition, a virtual object in the form of a virtual arrow 11 shows the user A, while going through the instruction, which area the user A must focus his attention on. Furthermore, a text field 15 provides the user A with text and, in particular, a note. The text contained in the text field 15 can be reproduced via the audio output unit 1d. Alternatively, a text different from the text displayed in the text field 15 can also be reproduced as an audio signal.

[0061] It is to be understood that the Fig. 2 The image shown on the display unit 1c is only a snapshot during the instruction process, and the individual virtual objects may change during the instruction process. Furthermore, virtual images can also be displayed during the instruction process without the support of the audio signal, and vice versa.

[0062] Fig. 3 shows one of the Fig. 1 corresponding embodiment of the AR system 1, wherein a monitoring unit 15 is additionally arranged in the real environment, which monitors a monitoring room 17. The work unit 3 and the security area 7 of the work unit 3 are located in the monitoring room 17. The monitoring unit 15 sends security data to the communication unit 1b of the AR system 1, which is processed by the control unit (not shown) of the AR system 1. Based on the security data of the monitoring unit 15, the control unit determines whether a user A is entering the monitoring room 17 and / or is located in the monitoring room 17. If the determination shows that a user A is entering the monitoring room 17, the audio output unit 1d of the AR system 1 outputs a first audio signal to notify the user A that he has entered the monitoring room 17.Additionally or alternatively, the work unit 3 can reduce its speed to reduce damage in the event of a potential collision with the user A. Furthermore, a second audio signal different from the first audio signal is emitted if the processing of the safety data by the control unit indicates that the user A is entering the safety area 7, which is also detected by the monitoring unit 19. Additionally or alternatively, the work unit 3 can terminate and stop the work operation in such a case.

[0063] Fig. 4 shows an embodiment of the AR system 1 in which the acquisition unit 1a is located in the immediate vicinity of the work unit 3, while the user A and the digital terminal 5, which comprises the communication unit 1b, the display unit 1c, and the audio output unit 1d, are not located in the immediate vicinity of the work unit 3. In such a case, the acquisition unit 1a and the digital terminal 5 or the communication unit 1b communicate with each other via a remote connection 19, e.g., a LAN, VPN, and / or mobile radio connection, so that the user A does not have to be physically on site or in the immediate vicinity of the work unit 3 to participate in an instruction. Consequently, by means of such an embodiment of the invention, teleinstruction is possible over any distance between the user A or the digital terminal 5 and the work unit 3 or the acquisition unit 1a.For example, an employee of a company can receive instruction from their private home using a digital device 5, with a camera on the company premises sending image and / or video data showing the work unit 3 located on the company premises to the digital device 5, on which user A receives the instruction. In such a case, even interactive group instruction—similar to a webinar—would be possible.

[0064] The aforementioned bidirectional interface, which allows certain sections to be repeated, could then only be performed by a "master"—for example, the instructor. The master ability can also be passed on to other participants. However, this can only be permitted for one user per time interval.

[0065] Fig. 5 shows an example representation of a user database that stores information about a user A of an application. The user database includes information about an employee number, a first name, a last name, a training level for a first application, a training validity period for a first application, a training level for a second application, and a training validity period for a second application. The training level of user A can be determined with a test, such as a multiple-choice test, which must be taken by the user after a training session has been completed in order to determine their corresponding training level.If user A answers more than a predetermined number of questions correctly after the instruction for a first application, the value "passed" or "1" is entered in the column "Instruction status regarding a first application", for example, while in all other cases the value "failed" or "0" is entered.

[0066] Furthermore, the validity period of the training indicates how many days a passed training is valid or in how many days a refresher training with regard to an application is necessary.

[0067] Additionally or alternatively, a qualification level for an application can also be defined in the user database. For example, a qualification level for a user A can be determined and entered into the user database based on the score achieved or the number of correctly answered questions in a test. Based on the qualification level, for example, different usage rights and / or access rights to content of AR System 1 can be defined for a user A. Such user databases with qualification levels can be stored in a central database of the respective company.

[0068] Fig. 6 shows different movement phases of the work unit 3 and an associated audio playback by the AR system 1 when a user A approaches the work unit 3. The AR system 1 functions in Fig. 6 as a warning system. For a better overview, Fig. 6 only the audio output unit 1d of the AR system 1 is shown.

[0069] In a first phase, user A with the AR system 1 is located outside the monitoring space 17 and the security area 7. The monitoring unit 15 (not shown) detects the monitoring space 17 and the security area 7 and sends security data to the AR system 1. The control unit of the AR system 1 processes the security data and determines that user A is outside the monitoring space 17 or that user A is not in the monitoring space 17. The audio output unit 1d consequently outputs an audio signal indicating that both critical areas are clear. Alternatively, the audio output unit 1d can also output no audio signal. In In a second phase, the AR system 1 determines that the user A has entered the monitoring room 17, but not the security area 7. As a result, a second audio signal is output, which indicates that the monitoring room has been entered and a reduced operating mode of the work unit 3 is active.

[0070] In In a third phase, the AR system 1 determines that the user A also enters the security area 7 and consequently outputs an audio signal via the audio output unit 1d, which indicates that the security area has been entered and the work unit 3 stops for safety reasons.

[0071] In In a fourth phase, user A leaves the security area 7 and remains in the monitoring room 17. The audio output unit 1d consequently outputs an audio signal indicating that the security area 7 has been left and the work unit 3 remains inactive.

[0072] In In a fifth phase, the AR system 1 determines that the user A has left both the security area 7 and the monitoring room 17 and consequently outputs an audio signal via the audio output unit 1d indicating that the monitoring room has been left and that the work unit 3 is still inactive and must be started manually. Bezugszeichenliste

[0073] 1 Augmented Reality System 1a Acquisition Unit 1b Communication Unit 1c Display Unit 1d Audio Output Unit 3 Work Unit 3a Controller 5 Digital Terminal 7 Security Area 9 Virtual Representation of the Work Unit 11 Virtual Arrow 13 Text Field 15 Monitoring Unit 17 Monitoring Room 19 Remote Connection

Claims

1. An augmented reality system (AR system) (1) for teaching a user (A) of an application, comprising: a detection unit (1a) which detects a real environment and provides it as environmental data, wherein at least one work unit (3) is arranged in the environment; a communication unit (1b) which receives process data from the work unit (3); a control unit which processes the process data and the environmental data, wherein a virtual image can be created based on the process data; a display unit (1c) which displays the real environment and the virtual image in relation to one another to a user; and an audio output unit (1d) which, on the basis of the process data, outputs audio signals for instructing a user (A), wherein the audio signals are preferably matched to the virtual image, wherein an instruction comprises one or a plurality of the following components: teaching a user, training a user, lecturing a user, assisting a user, instructing a user, specifying information with respect to a use of the work unit (3), specifying hazard warnings and / or warnings, and specifying general information, wherein the process data comprise data for identifying a process step of the work unit (3) and the audio output unit (1d) outputs an audio signal corresponding to an identified process step and / or the display unit (1c) displays a virtual image corresponding to an identified process step, wherein the AR system (1) comprises a memory in which a plurality of audio signal data and / or augmentation data are stored, wherein the audio signal data and augmentation data can be associated with the process steps and / or phases of a working life cycle and are categorized accordingly.

2. An AR system (1) according to claim 1, wherein the audio output unit is configured to play back the audio data and / or the display unit is configured to play back the augmentation data before a process step which will occur in the future and / or at the start of a process step.

3. An AR system (1) according to one of the claims 1 or 2, wherein the phases of a working life cycle comprise: putting into operation, normal operation, maintenance, conversion and / or dismantling of the application.

4. An AR system (1) according to any one of the preceding claims, wherein the AR system (1) and the work unit (3) are configured to communicate bidirectionally, wherein the AR system (1) is configured to control and / or to influence the work unit (3).

5. An AR system (1) according to any one of the preceding claims, wherein the display unit (1c) is additionally configured to display text messages.

6. An AR system (1) according to any one of the preceding claims, wherein the AR system (1) is configured to check an instruction level of the user (A) by means of a test, in particular a multiple choice test, and to enter and / or to update the instruction level of the user (A) in a user database based on a test result.

7. An AR system (1) according to claim 6, wherein the AR system (1) comprises an instruction level determination unit which is configured to determine an identity of the user (A) based on personal identification data and to determine the instruction level of the user (A) based on the determined identity by means of the user database.

8. An AR system (1) according to one of the claims 6 and 7, wherein the work unit (3) is configured to set an operating mode of the work unit (3) based on the instruction level of the user (A).

9. An AR system (1) according to any one of the claims 6 to 8, wherein the AR system (1) is configured to restrict or to extend a range of use and / or contents of the AR system (1) based on the instruction level of the user (A).

10. An AR system (1) according to any one of the preceding claims, wherein the real environment and / or the work unit comprises / comprise a marking which is used by the AR system (1) as a reference point and / or for recognizing the work unit (3).

11. An AR system (1) according to any one of the preceding claims, wherein a monitoring unit (15) is configured to monitor a monitored zone (17) and to transmit safety data to the communication unit (1b), wherein the safety data are processed by the control unit, and the audio output unit (1d) is configured to output an audio signal when the processing of the safety data reveals that the user (A) is entering the monitored zone (17) of the monitoring unit (15).

12. A method of teaching a user (A) of an application by means of an augmented reality system (1), said method comprising that: a real environment is detected and is provided as environmental data, wherein at least one work unit (3) is arranged in the environment; process data are received from the work unit (3); the process data and the environmental data are processed, wherein a virtual image is created based on the process data; the real environment and the virtual image are displayed in relation to one another to a user; and audio signals for instructing a user are output on the basis of the process data, wherein the audio signals are preferably matched to the virtual image, wherein an instruction comprises one or a plurality of the following components: teaching a user, training a user, lecturing a user, assisting a user, instructing a user, specifying information with respect to a use of the work unit (3), specifying hazard warnings and / or warnings, and specifying general information, wherein the process data comprise data for identifying a process step of the work unit (3) and the audio output unit (1d) outputs an audio signal corresponding to an identified process step and / or the display unit (1c) displays a virtual image corresponding to an identified process step, wherein the AR system (1) comprises a memory in which a plurality of audio signal data and / or augmentation data are stored, wherein the audio signal data and augmentation data can be associated with the process steps and / or phases of a working life cycle and are categorized accordingly.

13. A digital end device (5) for an augmented reality system (1) for teaching a user (A) of an application, said digital end device (5) comprising: a communication unit (1b) which receives process data from at least one work unit (3) and environmental data from at least one detection unit (1a) which detects a real environment, wherein the work unit (3) is arranged in the environment; a control unit which processes the process data and environmental data, wherein a virtual image can be created based on the process data; a display unit (1c) which displays the real environment and the virtual image in relation to one another to a user (A); and an audio output unit (1d) which, on the basis of the process data, outputs audio signals for instructing a user, wherein the audio signals are preferably matched to the virtual image, wherein an instruction comprises one or a plurality of the following components: teaching a user, training a user, lecturing a user, assisting a user, instructing a user, specifying information with respect to a use of the work unit (3), specifying hazard warnings and / or warnings, and specifying general information, wherein the process data comprise data for identifying a process step of the work unit (3) and the audio output unit (1d) outputs an audio signal corresponding to an identified process step and / or the display unit (1c) displays a virtual image corresponding to an identified process step, wherein the AR system (1) comprises a memory in which a plurality of audio signal data and / or augmentation data are stored, wherein the audio signal data and augmentation data can be associated with the process steps and / or phases of a working life cycle and are categorized accordingly.