Method and apparatus for linking real actuators with virtual objects

Abstract

The invention relates to an arrangement and method for the data linking of real actuators of technical functions of a device (12) with computer-generated virtual objects (O), said actuators being able to be actuated by an operator (15) by means of a control lever (1) and switch elements (6) arranged on a switch panel (7), comprising a simulation computing unit (11) for creating the virtual objects (O) and a portable reproduction unit (14) for showing the virtual objects (O) created by the simulation computing unit (11) in the field of view (B) in a real environment, said field of view being perceivable by the operator (15). It is proposed that by means of actuation sensors (3) for the switch elements (6), an actuation of a switch element (6) is detected and a corresponding switching signal is sent to a first position sensor (2), and the first position sensor (2) detects the setting of the switching lever (1) and sends a corresponding setting signal and the switching signal to the simulation computing unit (11), and the simulation computing unit (11) simulates the technical functions of the actuators as virtual objects (O) depending on the setting signals and the switching signals.

Description

[0001] Method and device for linking real actuators with virtual objects

[0002] The invention relates to an arrangement for the data-based linking of real actuators of technical functions of a device, which can be actuated by an operator by means of a control lever and switching elements arranged on a switch panel, with computer-generated, virtual objects, comprising a simulation computing unit for creating the computer-generated, virtual objects and a playback unit connected to the simulation computing unit via a signal connection for displaying the computer-generated, virtual objects created by the simulation computing unit in the field of view of a real environment perceived by the operator, in particular in the field of view of transparent data glasses, according to the preamble of claim 1. The invention further relates to a corresponding method for the data-based linking of real actuators of technical functions of a device with computer-generated, virtual objects, according to the preamble of claim 6.

[0003] The present invention concerns an application of so-called "mixed reality (MR)" systems or "augmented reality (AR)" systems. In this case, a virtual object computer-generated by a simulation processing unit is superimposed into the field of vision of a real environment perceived by a person, depending on the person's current position and viewing direction. For the determination of these positions, various sensors such as acceleration sensors (accelerometers or G-sensors), sometimes in combination with magnetometers and / or gyroscopes, as well as GPS systems, are conventionally used. These sensors are generally arranged on the display unit. Data glasses, i.e. devices that are worn like glasses and are capable of displaying virtual objects in the real world and positioning these objects in relation to real objects in space, are used in particular as display units.For this purpose, the "virtual coordinate system" of the virtual scenery is converted into the display coordinate system of the playback unit using an appropriate transformation matrix. Furthermore, it is known to equip playback units with a camera whose Cartesian coordinate system is also harmonized with the virtual coordinate system and the coordinate system of the playback unit. The position and spatial orientation of real objects relative to the world coordinate system of the real environment can subsequently be converted into the virtual coordinate system using optical and sensor-based tracking methods. Such data glasses are also referred to as "augmented reality" glasses or "mixed reality" glasses. A person wearing such glasses sees the environment as if through normal glasses, but (holographically virtual) objects can be superimposed into their field of vision.Alternatively, the use of lenses as a display unit is also conceivable, allowing virtual objects to be superimposed into the user's field of view. Various methods and devices for further developing such AR systems have been described, for example, in the applicant's Austrian patent AT 518,667, the applicant's European patent EP 3776148 B1, or the applicant's European patent applications EP 19709693.6 and EP 21735597.3.

[0004] One promising application is the education and training of civilian and military emergency personnel, such as the police, fire department, or the military. Emergency personnel must sometimes learn how to operate equipment such as special vehicles and other equipment, for example, while moving around a training ground and reacting correctly in various scenarios. Within the scope of the present invention, actuators for the technical functions of these devices are also discussed. These actuators can be actuating cylinders and ignition systems, for example, for positioning a tank barrel and firing a shot, or actuating cylinders and pumps for positioning an extinguishing agent ejector and ejecting extinguishing agent from an airport fire engine.In practice, training of this kind can be very complex and expensive, for example when soldiers are required to learn how to operate and fire a tank in combat, or when airport firefighters are required to extinguish an aircraft.

[0005] Therefore, simulators have been proposed in which the control cockpit for a vehicle is replicated, but the trainee moves entirely in a virtual world. The disadvantage of such simulators, however, is that they can only replicate reality to a limited extent, particularly with regard to the real-life requirements of having to move around a battlefield with a vehicle such as a tank in noise, vibration, and dust, and firing an accurate shot—which is precisely the goal of such training.

[0006] For the education and training of emergency personnel, it would therefore be best if the trainee could move through a real training area with a real device, simulating a variety of scenarios in which the trainee could practice mastering the device. A variety of scenarios could be simulated by displaying virtual objects.

[0007] The aim of the invention is therefore to provide devices and methods with which, for example, the technical devices used for the education and training of civilian and military forces can be linked to AR systems using data technology. The education and training of civilian and military forces, for example, could be improved and made more cost-effective in this way.

[0008] This objective is achieved by the features of claim 1. Claim 1 relates to an arrangement for the data-technical linking of real actuators of technical functions of a device, which can be actuated by an operator by means of a control lever and switching elements arranged on a switch panel, with computer-generated, virtual objects, comprising a simulation processing unit for creating the computer-generated, virtual objects and a playback unit connected to the simulation processing unit via a signal connection for displaying the computer-generated, virtual objects created by the simulation processing unit in the field of vision of a real environment perceived by the operator, in particular in the field of vision of transparent data glasses. According to the invention, it is proposed that a first position sensor for the control lever and actuation sensors for the switching elements are provided,wherein the actuation sensors are connected to the first position sensor via a signal line and are designed to detect an actuation of a switching element and to send a corresponding switching signal via the signal line to the first position sensor, and the first position sensor is connected to the simulation computing unit via a first data connection and is designed to detect the position of the gearshift lever and to send a corresponding actuating signal and the switching signal to the simulation computing unit, and the simulation computing unit is designed to simulate the technical functions of the actuators as computer-generated, virtual objects for insertion into the field of vision of a real environment perceived by the operator depending on the actuating signals and the switching signals, wherein the actuation sensors comprise a flat carrier with at least one surface area whose shape corresponds to the surface shape of the switch panel,and on which sensor elements are arranged in positions that correspond to the positioning of the switching elements on the switch panel.,

[0009] The invention thus relates to devices in which certain functions of the device are controlled via a control lever and switching elements arranged on the switch panel. Various actuators for technical functions, such as actuating cylinders, ignition systems, pumps, servomotors, and the like, are operated using the control lever ("joystick") and the switching elements. According to the invention, a first position sensor is provided for detecting the actuating movements of the control lever. This first position sensor detects the spatial position of the control lever, which conventionally generates the actuating signal for an actuator of the device. This position sensor is attached to the control lever of the device. It is therefore not necessary to intervene in the device's control system to obtain the actuating signals; instead, the positions of the control lever are specifically detected by the inventive arrangement, and the actuating signals are derived therefrom.The position sensor can be designed as an IMU (Inertial Measurement Unit). IMUs are known per se and enable the detection and measurement of translations and rotations using inertial sensors such as acceleration sensors and yaw rate sensors. The current position of the control lever can be calculated from these measured values. Suitable acceleration sensors for the inventive arrangement include, for example, conventional sensors with a resolution of 16 bits and a sampling rate of 400 Hz. Suitable yaw rate sensors for the inventive arrangement include, for example, conventional gyroscopes with a resolution of 16 bits and a sampling rate of 400 Hz. To improve measurement accuracy and correct long-term drift, magnetic field sensors are also usually used in IMUS.

[0010] The actuation sensors detect actuation of the switching elements, as will be explained in more detail below. Since there are a multitude of different switching elements, such as pushbutton switches, toggle switches, rocker switches, contact switches, slide switches and the like, the design of the actuation sensors will also vary, with each switching element being assigned an actuation sensor. If multiple switching elements are used, a corresponding number of actuation sensors are provided. If different switch types are used, actuation sensors of different designs can also be provided. However, these switching elements are usually arranged in close proximity to one another to enable quick and easy operation. The area over which these switching elements are distributed is also referred to as the switch field.

[0011] The inventive architecture further provides that the actuation sensors are connected to the first position sensor via a signal line, and the first position sensor is connected to the simulation computing unit via a first data connection. For this purpose, the first position sensor is provided with interfaces for the signal lines, which are designed, for example, as USB interfaces. Measurements from the actuation sensors can be streamed in real time via these interfaces. The first data connection between the position sensor and the simulation computing unit is preferably wireless, for example via a WLAN. If multiple control levers, each with a position sensor, are used, the position sensors can be synchronized wirelessly by providing all measurements with a time stamp from a common clock. For this purpose, the position sensors are connected to a common router.Such a synchronized network can be implemented, for example, with a WLAN router and USB charging hubs.

[0012] According to the invention, the simulation computing unit is designed to simulate the technical functions of the actuators as computer-generated, virtual objects for superimposing into the operator's field of vision of a real environment, depending on the control signals and the switching signals. This not only makes it possible to simulate the technical function itself—for example, projecting the raising and lowering of a tank barrel or the firing of a shot as virtual objects into the operator's field of vision—but also to simulate the interaction of these technical functions with real or virtual objects, for example, the detonation of a virtual shot at a virtual target or the extinguishing of virtual flames on an aircraft using virtual extinguishing agents.The arrangement according to the invention thus serves to link real actuators of a device's technical functions with computer-generated, virtual objects, allowing technical functions of the device such as the firing of a tank or the ejection of extinguishing agent from a fire engine to be simulated as virtual objects. The operator can handle the real device under real operational conditions, and the control of the device itself remains unchanged. However, in the above-mentioned examples of a tank, for example, no real firing of a shot would be required, or in the case of an airport fire engine, no real extinguishing agent would be required.

[0013] With regard to the actuation sensor, it is proposed that the sensor elements each be membrane switches, and that the flat carrier be designed as a PCB circuit board provided with recesses at the positions of the sensor elements. Such a design is simple and cost-effective to implement and is suitable for largely flat switch panels, so that the flat carrier needs no or only slight curvature because PCB circuit boards are largely rigid. At the positions of the sensor elements, which according to the invention correspond to the positions of the switching elements, the circuit board is provided with recesses. If the switching elements are, for example, pushbutton switches, the pushbutton switches can be actuated without restriction via the recesses in the circuit board.The membrane switches are located at the recesses, covering the recesses so that when a push button is pressed, the membrane switch is also actuated. Membrane switches are available in various designs and are generally made using multiple layers of film. A separating film and a covering film are arranged on a base film, and when pressed, conductor tracks on the separating film are electrically contacted by conductor tracks on the base film. The flat carrier is adapted to the surface shape of the switch panel in terms of size and shape with at least one surface area and is attached to the switch panel. In this operating position, the sensor elements are arranged in this surface area in positions that correspond to the positioning of the switching elements on the switch panel.

[0014] Alternatively, it is proposed that the sensor elements are each pressure sensors, and that the flat carrier is made of a flexible material to which the pressure sensors are attached. Such a design is suitable for switch panels with a strong curvature, for example for switch panels provided on the control lever itself. The flexible material can be a textile, for example, so that the flat carrier is a textile carrier. Textile carriers are sheet-like textile structures such as woven, knitted, crocheted, braided, stitch-knitted, nonwovens or felt, or three-dimensional structures made from them, which are manufactured from textile raw materials such as natural fibers or chemical fibers. With regard to such textile carriers, it is known to equip them with electronic components, for example with sensors for temperature or pressure, which are distributed across the textile carrier.Textile carriers equipped with such electronic components are also referred to as "smart textiles." These electronic components are designed as units that are connected to power sources or other components via electrical lines extending across the textile carrier. Within the scope of the arrangement according to the invention, pressure sensors can thus be arranged at the positions of the switching elements on the textile carrier, which is subsequently attached to the switch panel, for example, by being stretched over the switch panel. Due to the flexibility of the flat carrier, in this position of use, its shape corresponds to the surface shape of the switch panel in its surface area stretched over the switch panel. In this position of use, the sensor elements are arranged in this surface area in positions that correspond to the positioning of the switching elements on the switch panel.

[0015] The first position sensor determines the spatial position of the control lever relative to the world coordinate system, so that the control signals determined in this way can only be used directly if the reference system of the control lever is stationary relative to the world coordinate system. This is the case, for example, if the control lever is located in a stationary control panel from which an external device is controlled, or if the control lever is located on the device but the device is stationary during operation. If the control lever is located on a device that is moving during operation, for example, on an emergency vehicle, the absolute position of the control lever relative to the world coordinate system changes simply by changing the spatial position of the moving device.Therefore, in this case, it is proposed that a second position sensor be provided for the device, which is connected to the simulation computing unit via a second data connection and is designed to detect the spatial position of the device and send a corresponding correction signal for the control signal of the first position sensor to the simulation computing unit. The control signal relevant for controlling the device's activators is thus the differential signal from the first and second position sensors. The second data connection between the second position sensor and the simulation computing unit is also preferably implemented wirelessly, for example, via WLAN.

[0016] The invention further relates to a corresponding method for the data-technical linking of real actuators of technical functions of a device, which can be actuated by an operator by means of a control lever and switching elements arranged on a switch panel, with computer-generated, virtual objects by means of a simulation computing unit for creating computer-generated, virtual objects, and a playback unit for displaying the computer-generated, virtual objects in the field of vision of a real environment perceived by the operator, in particular in the field of vision of transparent data glasses.According to the invention, it is proposed that actuation of a switching element is detected by means of actuation sensors for the switching elements and a corresponding switching signal is sent via a signal line to a first position sensor for the control lever, and the first position sensor detects the position of the switching lever and sends a corresponding actuating signal and the switching signal to the simulation computing unit connected to the first position sensor via a first data connection, and the simulation computing unit simulates the technical functions of the actuators as computer-generated, virtual objects depending on the actuating signals and the switching signals and, with the aid of the playback unit, displays them in the field of vision of a real environment perceived by the operator.

[0017] In the event that the control lever is arranged on a device that is moving during operation, it is again proposed that the spatial position of the device is detected by means of a second position sensor for the device that is connected to the simulation computing unit via a second data connection and that a corresponding correction signal for the control signal of the first position sensor is sent to the simulation computing unit.

[0018] The invention will be explained in more detail below using exemplary embodiments with the aid of the accompanying drawings.

[0019] Fig. 1 shows a possible embodiment of a control lever provided with the arrangement according to the invention, and

[0020] Fig. 2 is a schematic representation of an application of the arrangement according to the invention.

[0021] Fig. 1 shows an embodiment of a control lever 1 for a technical device 12, for example an airport fire engine. In conventional operation, the control lever 1 controls actuators for technical functions of the device 12, for example actuating cylinders for the position of an extinguishing agent ejector 13 of the airport fire engine (see also Fig. 2). In this example, the technical function would be to adjust the spatial position of the extinguishing agent ejector 13. In the embodiment shown in Fig. 1, four switching elements 6.1, 6.2, 6.3 and 6.4, each of different designs, are arranged on the control lever 1; in the embodiment shown, however, all of them are actuated by pressing a button. A switching element 6.3 is designed, for example, as a push-button switch, which in conventional operation actuates an actuator designed as a pump, so that extinguishing agent is ejected via the extinguishing agent ejector 13.The technical function of this actuator in this case would be to eject an extinguishing agent. The number and design of the switching elements 6 can, of course, vary; for the purpose of explaining the invention, only the technical function of adjusting the extinguishing agent ejector 13 using the control lever 1 and the technical function of ejecting extinguishing agent using the switching element 6.3 will be discussed below.

[0022] The switching elements 6 are arranged on a switch panel 7 of the control lever 1. In the illustrated embodiment, the switch panel 7 is approximately planar, but the switching elements 6 are arranged spatially close to one another. To detect the actuation of the switching elements 6, an actuation sensor 3 is provided with a flat carrier 9 made of a flexible material, to which pressure sensors are attached as sensor elements 8 for the switching elements 6. Each of the switching elements 6.1, 6.2, 6.3, and 6.4 is assigned a sensor element 8.1, 8.2, 8.3, and 8.4. The flexible material can be a textile, for example, so that the flat carrier 9 is a textile carrier. As mentioned, textile carriers are sheet-like textile structures such as woven fabrics, knitted fabrics, braids, stitch-bonded fabrics, nonwovens or felts, and spatial structures made from them, which are made from textile raw materials such as natural fibres or chemical fibres.The sensor elements 8 are distributed over the textile carrier and connected to power sources or other components via electrical lines that extend over the textile carrier. The flat carrier 9 of the actuation sensor 3 can be attached to the switch panel 7, for example by being stretched over the switch panel 7 and adhering to the control lever 1 through tension elasticity. In Fig. 1, the actuation sensor 3 is shown separated from the control lever 1 for the sake of clarity of the drawing, but in the position of use the actuation sensor 3 is in contact with the upper area of ​​the control lever 1 and in particular with the switch panel 7. Due to the flexibility of the flat carrier 9, in this position of use its shape corresponds to the surface shape of the switch panel 7 in its surface area stretched over the switch panel 7 because it adapts to this.In this operating position, the sensor elements 8.1, 8.2, 8.3, and 8.4 are arranged in this surface area in positions that correspond to the positioning of the switching elements 6.1, 6.2, 6.3, and 6.4 on the switch panel 7, so that the sensor elements 8.1, 8.2, 8.3, and 8.4 detect the actuation of a switching element 6.1, 6.2, 6.3, and 6.4. However, the operability of the control lever 1 is not impaired in any way, because the switching elements 6.1, 6.2, 6.3, and 6.4 can still be actuated and thus control the actuators.

[0023] In the embodiment shown, only one actuation sensor 3 is shown, but several actuation sensors 3 could also be provided if the device is provided with several switch fields 7.

[0024] If the actuation of a switching element 6.1, 6.2, 6.3, and 6.4 is detected, a corresponding switching signal is sent from the actuation sensor 3 to a first position sensor 2. For this purpose, the actuation sensor 3 is connected via a signal line 10 to the first position sensor 2, which is attached to the control lever 1. The first position sensor 2 is provided with interfaces for the signal line 10, which are designed, for example, as USB interfaces. Measurements from the actuation sensor 3 can be streamed in real time via these interfaces.

[0025] The first position sensor 2 is attached to the control lever 1 and can be designed as an IMU (Inertial Measurement Unit). IMUs are known per se and enable the detection and measurement of translations and rotations using inertial sensors such as acceleration sensors and yaw rate sensors in all three spatial directions, as indicated by the three arrows in Fig. 1. From these measured values, the current position of the control lever 1 can be calculated. Conventional sensors with a resolution of 16 bits and a sampling rate of 400 Hz are suitable as acceleration sensors. Conventional gyroscopes with a resolution of 16 bits and a sampling rate of 400 Hz are suitable as yaw rate sensors for the arrangement according to the invention. To improve the accuracy of the measurement and to correct for long-term drift, magnetic field sensors are also commonly used in IMUs.With the help of the first position sensor 2, the actuating movements of the control lever 1 are detected, which in the conventional way generate the actuating signal for an actuator of the device 12. It is therefore not necessary to intervene in the control system of the device 12 in order to receive the actuating signals, for example, from an internal processor of the device 12. Instead, the positions of the control lever 1 are specifically recorded by the first position sensor 2, and the actuating signals are obtained from them. All that is required is knowledge of how an actuating movement of the control lever 1 is converted into a actuating signal, which is generally known. Control levers 1 usually function via an incremental control, i.e., for example, a forward actuating movement of the control lever 1 causes an actuation of an actuator in increments, which continues until the control lever 1 is returned to a neutral position.A larger actuating angle of the control lever 1 results in a faster incremental actuation of the actuator, while a smaller actuating angle results in a slower incremental actuation of the actuator. Both actuating data of such a control lever 1, i.e., the actuating angle and the duration of the actuating movement, can be obtained as a actuating signal via the first position sensor 2.

[0026] The actuating signal and any switching signals are subsequently sent from the first position sensor 2 via a first data connection 4 to an external simulation computing unit 11. The first data connection 4 is preferably wireless, for example via a WLAN. The simulation computing unit 11 is designed to simulate the technical functions of the actuators of the device 12 as computer-generated, virtual objects 0 depending on the actuating signals and the switching signals and to display these technical functions as virtual objects 0 in the field of view B of a real environment perceived by the operator 15 using the display unit 14 (see also Fig. 2). For the simulation of the technical functions, either data from the manufacturer of the technical device 12 can be used, from which the implementation of a

[0027] control and switching signals into corresponding technical functions, or the implementation is calibrated to a certain extent by linking measured control and switching signals with observed technical functions of the device 12.

[0028] The first position sensor 2 determines the spatial position of the control lever 1 relative to the world coordinate system, so that the control signals thus determined can only be used directly if the reference system of the control lever 1 is stationary relative to the world coordinate system. This is the case, for example, if the control lever 1 is arranged in a stationary control panel from which an external device 12 is controlled, or if the control lever 1 is arranged on the device 12, but the device is stationary during operation. If the control lever 1 is arranged on a device 12 that is moving during operation, for example, on a fire engine, the absolute position of the control lever 1 relative to the world coordinate system changes simply due to the change in the spatial position of the moving device 12.Therefore, a second position sensor 5 is provided, which can be attached to the device 12 and is connected to the simulation computing unit 11 via a second data connection 16. This second position sensor is designed to detect the spatial position of the device 12 and to send a corresponding correction signal for the control signal of the first position sensor 2 to the simulation computing unit 11. The control signal relevant for controlling the activators of the device 12 is thus the difference signal from the first position sensor 2 and the second position sensor 5. The second position sensor 5 is also designed as an IMU ("Inertial Measurement Unit"). Conventional sensors with a resolution of 16 bits and a sampling rate of 400 Hz are suitable as acceleration sensors. Conventional gyroscopes with a resolution of 16 bits and a sampling rate of 400 Hz, for example, are suitable as rotation rate sensors.Magnetic field sensors can also be used to improve measurement accuracy and correct long-term drift. The second data connection 16 is also preferably wireless, for example via a WLAN. The generation and reproduction of a virtual object 0 by the simulation computing unit 11 is generally known. Within the scope of the invention, the simulation computing unit 11 simulates the technical functions of the actuators as computer-generated, virtual objects 0 depending on the control signals and the switching signals and, with the aid of the reproduction unit 14, overlays them into the field of view B of a real environment perceived by the operator 15, as will be explained below with reference to Fig. 2.For this purpose, the simulation computing unit 11 is connected via the first data connection 4 to the first position sensor 2 for the control lever 1, via the second data connection 16 to the second position sensor 5 for the device 12, and via a signal connection 17 to the playback unit 14. The signal connection 17 is also preferably wireless, for example via a WLAN. In Fig. 2, the playback unit 14 is designed as a playback unit 14 that can be worn on the head by the operator 15, for example in the form of data glasses. However, at the time of registration, an increasing miniaturization of the playback unit 14 is observable, so that it cannot be ruled out that it could also be designed as a lens that can be worn in the eye in the future.

[0029] In the example of an airport firefighting vehicle shown in Fig. 2, the control signals are, for example, control signals for the extinguishing agent ejector 13, which thus determine the positioning of the extinguishing agent ejector 13. In practical use, an operator 15 could sit in the firefighting vehicle and, using the control lever 1, adjust the extinguishing agent ejector 13 in the conventional manner and also operate the corresponding actuators so that the extinguishing agent ejector 13 actually moves. However, the simulation computing unit 11 also has data on the current positioning of the extinguishing agent ejector 13 via the first position sensor 2. The operator 15 can also actuate the switching elements 6.1, 6.2, 6.3 and 6.4 without restriction and, for example, trigger the technical function of ejecting extinguishing agent by actuating the switching element 6.3.During a simulation, however, the corresponding pumps are de-energized, or there is no extinguishing agent on board the fire engine, so no physical extinguishing agent is ejected. Instead, the simulation computing unit 11 calculates the trajectory for the extinguishing agent from the known control signals and, upon detection of the corresponding switching signal from switching element 6.3, displays the extinguishing agent as a computer-generated, virtual object 0 in the field of view B of the real environment, for example, in the field of view of transparent data glasses.

[0030] Furthermore, it is also possible to simulate an interaction of this technical function of ejecting an extinguishing agent displayed as a computer-generated, virtual object 0 with real or virtual objects, for example, the extinguishing of virtual flames on a real or virtual aircraft. The arrangement according to the invention thus serves to data-technically link real actuators of technical functions of a device 12 with computer-generated, virtual objects 0, so that technical functions of the device 12, such as the ejection of extinguishing agent from a fire engine, can be simulated as virtual objects 0. The operator 15 can handle the real device 12 under real operating conditions, and the control of the device 12 itself is not changed.

[0031] The invention thus provides a device and a method with which, for example, the technical devices 12 used for the education and training of civilian and military forces can be linked to AR systems using data technology. The education and training of civilian and military forces, for example, can be improved and carried out more cost-effectively in this way.

Claims

Patent claims:

1. Arrangement for the data-technical linking of real actuators of technical functions of a device (12) operable by an operator (15) by means of a control lever (1) and switching elements (6) arranged on a switch panel (7) with computer-generated, virtual objects (0), comprising a simulation computing unit (11) for creating the computer-generated, virtual objects (0) and a display unit (14) connected to the simulation computing unit (11) via a signal connection (17) for displaying the computer-generated, virtual objects (0) created by the simulation computing unit (11) in the field of view (B) of a real environment perceived by the operator (15), in particular in the field of view of transparent data glasses, characterized in that a first position sensor (2) for the control lever (1) and actuation sensors (3) for the switching elements (6) are provided,wherein the actuation sensors (3) are connected to the first position sensor (2) via a signal line (10) and are designed to detect an actuation of a switching element (6), :and to send a corresponding switching signal via the signal line (10) to the first position sensor (2), and the first position sensor (2) is connected to the simulation computing unit (11) via a first data connection (4) and is designed to detect the position of the gearshift lever (1) and to send a corresponding actuating signal and the switching signal to the simulation computing unit (11), and the simulation computing unit (11) is designed to simulate the technical functions of the actuators as computer-generated, virtual objects (0) for fading into the field of view (B) of a real environment perceived by the operator (15) as a function of the actuating signals and the switching signals, wherein the actuation sensors (3) comprise a flat carrier (9) with at least one surface area, the shape of which corresponds to the surface shape of the switch panel (7), and on which sensor elements (8) are arranged in Positions are arranged which correspond to the positioning of the switching elements (6) on the switch panel (7).

2. Arrangement according to claim 1, characterized in that the sensor elements (8) are each membrane buttons, and the flat carrier (9) is designed as a PCB circuit board which is provided with recesses at the positions of the sensor elements (8).

3. Arrangement according to claim 1, characterized in that the sensor elements (8) are each pressure sensors, and the flat carrier (9) is made of a flexible material to which the pressure sensors are attached.

4. Arrangement according to one of claims 1 to 3, characterized in that a second position sensor (5) is provided for the device (12), which is connected to the simulation computing unit (11) via a second data connection (16) and is designed to detect the spatial position of the device (12) and to send a corresponding correction signal for the actuating signal of the first position sensor (2) to the simulation computing unit (11).

5. Device (12) with an arrangement according to one of claims 1 to 4.

6. Method for the data-technical linking of real actuators of technical functions of a device (12) which can be actuated by an operator (15) by means of a control lever (1) and switching elements (6) arranged on a switch panel (7) with computer-generated, virtual objects (0) by means of a simulation computing unit (11) for creating computer-generated, virtual objects (0), and a playback unit (14) for displaying the computer-generated, virtual objects (0) in the field of view (B) of a real environment perceived by the operator (15), in particular in the field of view of transparent data glasses, characterized in that by means of actuation sensors (3) for the switching elements (6), an actuation of a switching element (6) is detected and a corresponding switching signal is sent via a signal line (10) to a first position sensor (2) for the control lever (1), and the first position sensor (2) detects the position of the switching lever (1) and sends a corresponding actuating signal and the switching signal to the simulation computing unit (11) connected to the first position sensor (2) via a first data connection (4), and the simulation computing unit (11) simulates the technical functions of the actuators as computer-generated, virtual objects (0) depending on the actuating signals and the switching signals and, with the aid of the reproduction unit (14), overlays them into the field of vision (B) of a real environment perceived by the operator (15).

7. Method according to claim 6, characterized in that by means of a second position sensor (5) for the device (12) connected to the simulation computing unit (11) via a second data connection (16), the spatial position of the device (12) is detected and a corresponding correction signal for the control signal of the first position sensor (2) is sent to the simulation computing unit (11).

Images