Monitoring a machine with predicted vision

Abstract

Device (4) for monitoring a real machine (6) located in a real environment (16), which can be controlled by a human operator (10) using a real control command (18), - with a control input (22) for inputting the control command (18), - with a machine observer module (24) which is configured to simulate a virtual machine model (26) of the machine (6) in real time based on the input control command (18), - with an image observer module (28) which is configured to simulate a virtual image model (30) of the environment (16) of the machine (6) in real time based on the virtual machine model (26) simulated in real time and to output the virtual image model (30) simulated in real time to a display module (14), in particular the device (4), for display by means of the display module (14) in real time.

Description

State of the art

[0001] The monitoring of autonomous machines, particularly vehicles, is carried out by human personnel / operators. If a situation or task cannot be performed automatically, manual intervention by personnel is necessary. This can be done on-site (the operator must reach the machine, e.g., a driver must reach a vehicle) or remotely from a control center. This changes the personnel's role from simply monitoring the machine / vehicle to operating it (operator, teleoperator). Human-machine interface (HMI) concepts (Direct Control) for the teleoperator are known, which mimic the vehicle's / machine's HMI. With more abstract operating concepts, the teleoperator specifies, for example, a path / trajectory to be followed (Indirect Control).

[0002] Furthermore, methods are known that enable deterministic latency in data transmission between the vehicle / machine and the teleoperator. This means that data transmission, e.g., the transmission of a video image / frame showing the machine and / or its surroundings, occurs after a maximum or precisely defined latency (e.g., 300 ms).

[0003] DE 10 2020 121 561 B3 discloses a robot control system and a display device using augmented reality and mixed reality.

[0004] DE 10 2019 006 800 A1 discloses a control method for a bilateral teleoperation system.

[0005] EP 3 572 914 A2 discloses interactive two-way 3D real-time operations of virtual 3D real-time objects in a virtual 3D real-time world. US 2022 / 0 184 803 A1 discloses a system and methods for creating an immersive virtual environment with a virtual reality system that receives parameters corresponding to a real robot.

[0006] US 2021 / 0 094 180 A1 reveals systems and methods for human-robot communication.

[0007] US 2021 / 0 018 914 reveals an autonomous robot with on-demand human intervention. Disclosure of the invention

[0008] Against this background, the objective is to provide a device and a method for monitoring a real machine located in a real environment, as well as a corresponding computer program and a computer-readable medium, which enable the most realistic possible operation of the machine and a detailed representation of the machine's environment. This objective is achieved by the subject matter of the invention, namely a device according to claim 1, a method according to claim 11, a computer program according to claim 12, and a computer-readable medium according to claim 13. Preferred or advantageous

[0009] Embodiments of the invention and other categories of invention (computer program, computer-readable medium) are described in the further claims, the following description and the accompanying figures.

[0010] The device is designed for monitoring a real machine. The machine is located in a real environment. The machine can be controlled by a human operator using a real control command, or is controlled by the operator. In other words, the invention presupposes that a corresponding situation exists, i.e., that the real machine is controllable by the operator, etc.

[0011] The device includes a control input. This input is designed to receive at least one control command, in particular control commands sent by the operator to the actual machine. In other words, the control input intercepts or monitors the control commands sent by the operator to the machine. The operator uses a human-machine interface (HMI), especially in a control room, to generate or output the control commands to the machine and the device. The HMI is used for monitoring and controlling the machine.

[0012] The system includes a machine observer module. This module is designed to simulate a virtual machine model of the machine in real time based on the control command(s). In other words, an observer is created which, within the scope of standard engineering practice, attempts to replicate the machine and its behavior / response to control commands in all aspects necessary for the respective application, or to virtually predict the behavior of the real machine. This is done in a standard engineering manner with the aid of various additional information, such as the necessary physical / dynamic characteristics of the machine and the environment in which the machine is currently located, etc.

[0013] The system also includes an image monitoring module. This module is designed to simulate a virtual image model of the machine's environment in real time, based on the machine model. The explanations for the machine model above apply analogously to the image model; that is, it is tracked in the usual way, acting as an observer, to virtually replicate, simulate, or predict the development of the real environment in response to the control commands of the real machine. In addition to the machine's environment, the image model also includes at least parts of, or even the entire, machine. The image model serves as a basis for subsequently creating virtual representations of the environment / machine within that environment.The image observer module is further configured to output the virtual image model simulated in real time to a display module, in particular a display module assigned to or arranged on the device (4), for display by means of the display module (14) in real time.

[0014] The system optionally includes a display module. This module is designed to generate and output the image model, or the virtual views / perspectives derived from it, of the environment / machine in real time. "Display" here is to be understood broadly and also includes the output of image data that can then be viewed outside the system, for example, on a control room monitor. The display module may, but does not necessarily have to, include a display device. The display is primarily intended for the operator, allowing them to view the image model or the views / images derived from it.

[0015] In other words, the display module shows a view of a virtual situation that depicts in real time how (according to predictions from the machine model and image model) the real machine is likely to behave in its real environment. In other words, it approximates or predicts a real-time representation of the machine and its surroundings. This enables real-time monitoring of the machine. Specifically, the (predicted / simulated) reaction of the machine to its environment can be observed in real time, allowing for instantaneous real-time control of the machine.

[0016] The machine is, in particular, a work machine or a vehicle, especially an autonomously operating and / or moving / driving machine. As described above, the control system serves, in particular, as an additional control or emergency control in the event that autonomous operation / driving of the machine is not desired or not possible.

[0017] For reasons of plausibility, the invention is partly explained using a vehicle as an example of a machine. However, the statements also apply analogously to other suitable moving or non-moving machines, particularly autonomous ones, that need to be monitored and possibly controlled.

[0018] The display module primarily shows / outputs a virtual working situation of the machine or a virtual driving state of the vehicle. Corresponding control commands include, in particular: steering, braking, accelerating a vehicle, raising or lowering the forks of a forklift, the bucket of an excavator, operating and swiveling states of a tractor implement, etc.

[0019] The display module or the virtual image model contains / shows in particular the environment of the machine with or without the machine, for example a bird's-eye view / a side view of a vehicle, a view of the environment of a vehicle in the direction of travel seen from the vehicle, etc.

[0020] In other words, a virtual view of the machine itself and / or from the machine, including its surroundings, is virtually generated / displayed. The terms "image" and "display" are to be understood in their broadest sense and therefore also include information / representations from any environmental sensors, such as distance sensors, radar sensors, image data in the visible or other spectral ranges, etc.

[0021] The phrase "based on" the control command / machine model means that the control command / machine model, either in its entirety or only partially, or parameters derived from it, are incorporated into the simulation. In other words, aspects / parameters of the control command / machine model are used in the simulation / prediction.

[0022] According to the invention, a method for monitoring and remote control of machines / vehicles is thus provided. This results in a model-based observer for real-time control in situations with limited data transmission performance between the control room / HMI and the machine, or data transmission from the machine or its environment back to the operator / monitor.

[0023] The invention is based on the following findings: Remote control of vehicles or mobile machinery is carried out by a teleoperator. The invention specifically assumes that there is no direct line of sight between the operator and the machine or vehicle. Therefore, the teleoperator is presented with image information / camera images from the vehicle's surroundings, which he uses to monitor and control / operate the machine. At high transmission rates, individual frames are lost during the transmission of image information. The method according to the invention achieves a continuous image sequence even when sporadic image losses occur, since the virtual model of the machine is available in the system and correspondingly simulated representations of the machine's real-time behavior are possible without requiring actual real-time data transmission from the machine to the operator.This improves the machine's usability for the teleoperator. The operator view (via display on the screen module) shows vehicle movements and reactions to control commands according to the natural vehicle behavior (represented by the virtual simulation).

[0024] According to the invention, latency (limited data transmission performance) between the machine and the teleoperator is compensated. A high-performance data connection between the vehicle and the operator is therefore unnecessary. This enables vehicle operation even in locations or at times with low transmission performance and reduces operating costs (reduced data volume and latency requirements). The method is equally applicable, in particular, to vehicles (PC: passenger car / CV: commercial vehicle) or mobile machinery (wheel loaders, forklifts, tractors, etc.).

[0025] According to the invention, the teleoperator is enabled to operate the machine as realistically as possible. In particular, a highly detailed representation of the machine's environment is made possible. This allows for a large field of view, high image resolution, and the recognition and display of objects. Vehicle movement is displayed in real time. The machine's pose (position, state, and behavior within its environment) is displayed without delay. High frame rates of typically 16 to 24 frames per second (fps) are possible for displaying the (virtual) environment, ensuring smooth motion. The invention enables the display / reproduction of the vehicle's or machine's immediate response to the operator's control commands.

[0026] The invention is based on the requirement that both the real vehicle and its virtual representation must respond to the teleoperator's commands immediately, i.e., with imperceptibly short delays. According to the invention, this results in tolerance for lower-performing data transmission, a reduction in the amount of data to be transmitted, and the acceptance of higher and non-constant latency.

[0027] "Real-time" here includes, in the usual technical sense, tolerable latencies. The maximum possible latency at which convenient remote control / monitoring "in real time" is still possible depends, for example, on the vehicle's speed, the dynamics of changing driving conditions (steering, braking, accelerating), and the field of view of the environmental sensors. "Real-time" should therefore be understood as meaning that at least the rendering of the virtual image model / its display occurs within the maximum tolerable latency.

[0028] The invention is particularly suitable for use in autonomous commercial vehicles and autonomous mobile work machines, which are currently projected to have a large market. However, the algorithms that enable this autonomy will not be able to autonomously handle all possible situations / tasks in the foreseeable future; this is the premise of the invention. In these cases, human intervention (operator) is necessary, ideally through remote access by a teleoperator via control commands. Teleoperation / remote control is mandatory for the type approval (homologation) of autonomous vehicles and is also verified during homologation testing. Since high-performance network coverage (remote control of the machine via a data network, feedback of information / image data from the environment) is not available everywhere, the method according to the invention is particularly advantageous in such situations.

[0029] According to the invention, a method for monitoring vehicles and machines (with predicted vision) is thus provided. This enables corresponding remote control of the machine using predicted vision. "Predicted vision" means the operator viewing the virtual image model on / with the aid of the display module.

[0030] According to one aspect of the invention, the device is configured to simulate the machine model using the machine observer module and the image model using the image observer module in real time and to display them using the display module within a delay time after input or receipt of the control command via the control input, wherein the delay time is at most a predefinable maximum time. In other words, according to one aspect of the invention, the device is configured for real-time operation (simulation of the models, provision / display on the display module) by the following measures: Within a delay time – from receipt of the control command at the control input – the machine model and the image model are simulated, and the image model is displayed / provided on the display module. The delay time is at most as long as a predefinable maximum time. This maximum time is, in particular, the maximum permissible delay explained above.Tolerable latency time between sending or receiving the control command in the machine and the operator's display / observation of the machine's reaction in the environment.

[0031] The invention is based on the understanding that the receipt of the control command by the device, which marks the start of the delay time, coincides with the moment the control command is sent to the machine. Any latency in the transmission from the operator to the machine can therefore be used to simulate the models and display the simulated response to the operator. As explained above, the predefined maximum time can be selected to suit the situation, depending on how time-critical the respective application / machine / its operation is.

[0032] According to one aspect of the invention, the predefinable maximum time is a single or multiple of the refresh rate of the display module / display to the operator. Particularly in the case of a single refresh rate, it is ensured that after the control command is issued or received by the device, the next image displayed to the operator by the display module is already generated based on the simulation according to the invention, thus showing the (earliest possible within the frame repetition) immediate representation of the virtual reaction of the machine / environment to the control command. The refresh rate results, in particular, from the time intervals between two individual images for typical frame rates of 16 to 24 frames per second.

[0033] According to one aspect of the invention, the device includes a parameter input for receiving a real machine parameter and a machine correction module. The machine correction module is configured to correct the machine model based on the machine parameter. The procedure for this correction is standard practice: Essentially, the current virtual machine model or its parameters are compared with the received real machine parameters and thus with the real machine, deviations are determined if necessary, and corrections are made to minimize or eliminate the deviations. Machine parameters are, in particular, status variables of the machine such as speed, direction of travel, acceleration, deceleration, or steering angle of a vehicle.Typically, dead-time elements are used to compensate for the runtimes between the virtual machine model and the real machine, ensuring that the comparison between the virtual and real machines always occurs at a specific, consistent point in time. This is because transmitting machine parameters from the real machine to the system can take a certain amount of time, depending on the latency of the data connection, etc. This ensures that the virtual machine model can be updated as closely as possible based on the real machine parameters, even if the predicted machine model deviates from reality.

[0034] According to one aspect of the invention, an imaging module is assigned to the machine. This module is configured to generate a real-world image of the machine's environment (and optionally, of part or all of the machine). In other words, the invention assumes that a corresponding imaging module is present and that corresponding machine images are generated. The device includes an image input that serves to receive the machine images. The image monitoring module is then configured to simulate the virtual image model of the machine's environment in real time, also based on the machine image. A corresponding imaging module is, for example, a front-facing camera mounted on a vehicle or a multi-camera system that maps the real-world environment of the real machine.The machine image is therefore, in particular, a real image of the machine's surroundings, possibly with the machine or from the machine, especially taken by a camera mounted on the machine.

[0035] This allows historical, but nonetheless real, image material of the environment / machine to be incorporated into the image model. For example, it is possible to create a particularly realistic virtual image model by fitting a virtual machine (predicting the machine in real time) into a real historical environment (a real historical machine image) according to current predictions. This results in a highly realistic virtual image model. The integration of real image elements thus enables, in particular, a mixed virtual / real display on the screen module.

[0036] In particular, it is also intended to detect moving objects (people, vehicles, machines, obstacles) in the vicinity of the remotely controlled machine through object recognition based on image information or using other environmental sensors such as ultrasound and radar. When these objects move, this movement is also predicted in the image monitoring module and can thus be displayed in real time in the image model by the display module.

[0037] According to one aspect of the invention, the device includes an image correction module. This module is configured to correct the image model based on the machine image. The correction is performed in the usual manner, as already explained above, by comparing and determining deviations between the virtual image model / its parameters and the real machine image, and by inserting corrections to minimize or eliminate these deviations. Here, too, as explained above, dead-time elements are used to compensate for the propagation delays between the virtual image model and the real machine image. As explained above, this also prevents the image model from deviating too much from reality: If necessary, a correction is made should the image model of the environment differ from the actual real environment of the machine.

[0038] According to one aspect of the invention, the device includes an image machine correction module. This module is configured to correct the machine model based on the machine image and, optionally, also on the machine parameter. The image machine correction module thus uses the machine image, and optionally also the machine parameter, to correct the machine model. The statements already made above apply analogously: the correction is carried out in the usual manner by comparison, identification of deviations, and, if necessary, correction to minimize the deviations / errors, etc. Corresponding dead time compensation is also carried out analogously as explained above. In this way, a particularly effective correction of the machine model can be achieved.

[0039] According to one aspect of the invention, the device includes environmental information, in particular a memory for such environmental information, and / or an information input for feeding such environmental information into the device. The device is configured to simulate the machine model and / or the image model also based on the environmental information. Such environmental information includes, for example, map information, 3D maps, information from surrounding cameras, fixed cameras in the environment in which the machine is located or moving, cameras on other machines, in particular vehicles, geoinformation / GPS information about the (current) location of the machine, etc.

[0040] By processing the relevant information, both the machine model and the image model, or their behavior, can be simulated / predicted even more accurately.

[0041] Another object of the invention is a machine system. This system includes the device according to the invention as well as the machine mentioned in this context. The machine system and at least some of its possible embodiments, as well as their respective advantages, have already been explained in substance in connection with the device according to the invention.

[0042] In particular, the machine system also includes the aforementioned elements, such as a module for generating or providing the machine's environmental information / this environmental information, the imaging module for generating the real machine image of the machine's environment, etc. Specifically, the machine / possibly a control room and the facility are connected via a data network for communication purposes, enabling the exchange of control commands, machine parameters, machine images, etc.

[0043] According to one aspect of the invention, the machine system incorporates this data network. The machine, a control room (if applicable), and the surrounding equipment are communicatively connected via this data network, as are any additional components mentioned above. However, the data network is, at least temporarily, unable to provide a real-time representation of the environment / machine. By "real-time representation," we mean one comparable to the virtual image model / display on the screen module, i.e., one that would provide the operator with corresponding real-world, rather than virtual, image information about the environment / machine.

[0044] This is precisely where the inventive effect comes into play, namely that the virtual image of the environment in the form of the image model can be displayed / output in real time on the display module and is thus available to the operator.

[0045] The invention is based on the following findings, observations, and considerations and further comprises the following preferred embodiments. These embodiments are sometimes referred to simply as "the invention." The embodiments may also include parts or combinations of the embodiments mentioned above, correspond to them, and / or may include previously unmentioned embodiments.

[0046] The invention assumes that control commands (especially motion control data for the machine) from the operator to the machine and corresponding feedback from the machine to the operator (machine parameters) are transmitted relatively frequently with short cycle times but generate only a small data volume. Extensive image data (real machine image of the environment / machine), on the other hand, are transmitted infrequently, i.e., with long cycle times and latency. To nevertheless present the operator with a smooth, real-time motion sequence of the machine, a model-based observer in the form of the machine model and the image model is used. For example, data flows in the case of remote control of a machine in the form of a vehicle by a teleoperator in the form of an operator are represented as follows: Control commands in the form of manipulated variables are transmitted from a control room (operator) to a machine (vehicle).Examples of relevant data include the position of the accelerator pedal, the brake pedal, and the steering wheel angle. Cycle times for this are 20 to 100 ms, with a data volume of 48 bits per cycle (this is just an example; the exact data volume may vary depending on the application). State variables (machine parameters) are transmitted from the vehicle to the control room, i.e., from the machine to the operator. These include, for example, speed, acceleration, yaw rate, drive torque, braking torque, and object coordinates of the vehicle's surroundings. Here, too, the cycle time is 20 to 100 ms. A typical data volume is, for example, 80 plus 768 bits (vehicle data plus coordinates of 16 objects). Image data transmitted from the machine to the control room (operator) includes, for example, video image data (depending on the movement status of a front or rear camera). Here, the cycle time is 500 to 1500 ms, and the data volume is 1.5 MB.The HMI transmits data from the control room (display module) to the operator, for example, by displaying it on a high-resolution screen. The data transmitted is an image of the vehicle's surroundings, including, for example, the actual image and additional object markers. At a refresh rate of 20 frames per second (fps), the cycle time is 50 ms; the data volume per cycle is, for example, 5 MB.

[0047] The device according to the invention, or a method carried out with the device, thus provides a virtual vehicle model (machine observer module, machine model) which determines / predicts the current vehicle pose based on the control data (control commands). The real state and motion data subsequently reported back by the actual vehicle are used for the subsequent correction of the vehicle model. A dead-time element with a time constant corresponding to the latency (dead time) is used for phase correction of observed (virtual models) and measured (real) values. Methods are known that can determine the runtime of the data transmission (control command via the network to the machine / machine parameters via the network to the parameter input, machine image via the network to the image input) via networks. Together with the processing time on the vehicle, this results in the total dead time in the signal path.This can then be taken into account statically (with a constant dead time) or preferably dynamically (according to the currently measured dead time).

[0048] The observed pose (predicted machine model, image model), which ideally corresponds to the actual vehicle pose, is passed to the image observer module for further processing.

[0049] A current view (virtual image model) is calculated from the most recently transmitted real-world image data (latest available machine image, historical data) and the predicted change in vehicle pose. As soon as new real-world image data is available, a subsequent comparison is performed in the image monitoring module (image correction module using the latest, albeit historical, machine image). The image data (machine image) transmitted by the imaging module (e.g., the vehicle's vision system) can be further processed by the image monitoring module for display on the HMI; for example, the viewing angle can be adjusted, etc.

[0050] The following image processing techniques are used to calculate new views (image model / display module) based on past image data (machine image) (using a front camera as an example): • Longitudinal vehicle movements are simulated by a digital zoom, even if this results in a loss of image resolution. • Transverse vehicle movement can be simulated by changing the image section; this narrows the field of view. Therefore, it is advantageous to present the operator with only a limited field of view when driving straight ahead, in order to avoid "black edges" when changing the image section.

[0051] In further advantageous embodiments, the machine's environmental data (image, objects, point coordinates) can be compared with 3D maps stored in the control room / facility. Corresponding SLAM (Simultaneous Localization and Mapping) methods are known. The vehicle's pose is determined by comparing the stored map data with the transmitted data. The image machine correction module then corrects the observed vehicle pose (virtual machine model).

[0052] To improve optical quality or expand the field of view, the following measures can optionally be applied: • Image information from the vehicle (machine image) can be supplemented with that of other vehicles or stationary cameras (environmental information) to create a particularly suitable view (image model) for the teleoperator. • Furthermore, it is possible to extend the teleoperator's field of view (image model) using the 3D map data. For this purpose, the map data is rendered according to the vehicle's pose (machine model) and the prevailing lighting conditions (known from the image data transmitted by the vehicle / machine image). This method requires up-to-date map data; for example, it must be ensured that other road users are not within the extended image area. Meeting this requirement is easily achievable in restricted areas (depot, mine, etc.).

[0053] Other possible alternatives of the invention are: • The vehicle pose can be checked by transferring the geoposition. • In cases of low vehicle dynamics or short latency, correction by the machine correction module can be omitted. • The transfer of object data is optional. • Nonlinear or (piecewise) linear observers can be used. • Measurement parameters other than those mentioned (machine parameters) can also be used to correct the observers (machine model, image model) (observability criteria must be met). • The teleoperator's HMI does not need to be modeled after the vehicle / machine HMI. • The machine observer module / image observer module can be executed synchronously or asynchronously (also in different time grids). • If the vehicle's imaging module has a 360° view, the image area does not need to be restricted when cornering.

[0054] Further features, effects, and advantages of the invention will become apparent from the following description of a preferred embodiment of the invention and the accompanying figures. These figures are shown in schematic diagrams: Fig. 1 a machine system according to the invention in a symbolic block diagram, Fig. 2. the temporal progression of various data and quantities in the machine system Fig. 1.

[0055] Fig. Figure 1 shows a machine system 2. This system includes a device 4 according to the invention and a machine 6, here in the form of an autonomously driving vehicle. The vehicle 6 moves in an environment 16, here a tunnel of a mine. The vehicle 6 is monitored via a control room 8. An operator 10, acting as a teleoperator, is located in the control room 8. Using a human-machine interface (HMI) 12, the operator can monitor the machine 6 and its environment 16, in which the machine 6 moves, via a display module 14, here a monitor. If the autonomous driving capability of the machine 6 is insufficient, the operator 10 can also send a control command 18 to the machine 6 via the HMI 12, here a steering, acceleration, or braking command. There is no line of sight between the operator 10 / control room 8, here at the surface, and the machine 6 / environment 16.

[0056] Machine system 2 includes a data network 20, which establishes a wireless connection between control room 8 and facility 4 on the one hand, and machine 6 and its surroundings 16 on the other. Control room 8, machine 6, and facility 4 are thus communicatively connected via data network 20. However, data network 20 is not capable of transmitting image data from the surroundings 16 to the display module 14 quickly enough for the operator 10 to monitor or control machine 6 with an acceptable delay via live video surveillance. For this reason, facility 4 is used.

[0057] Device 4 serves to monitor the real machine 6 located in the real environment 16, whereby the machine 6 can be controlled by the human operator 10 using the real control commands 18. Device 4 contains a control input 22 for inputting and receiving the control command 18, which is sent to the machine 6 by the operator 10. Device 4 contains a machine observer module 24. This module is configured to simulate a virtual machine model 26 of the real machine 6 in real time based on the control command 18. The machine model 26 is also called the observer. The machine model 26 virtually simulates the behavior of the machine 6 in response to the control commands 18 in the environment 16.

[0058] The device 4 also includes an image monitoring module 28. This module is configured to simulate a virtual image model 30 of the machine 6's environment 16 in real time, based on the machine model 26. The virtual image model 30 virtually replicates the environment 16 with the machine 6 located and moving within it, thus generating a virtual representation of the environment 16. The display module 14 is also part of the device 4 and is configured to display the image model 30, or views / images of the virtually replicated environment 16 derived from it, in real time, so that the operator 10 can view it.

[0059] As a result, the operator 10 sees a predicted simulation on the display module 14 in real time, showing how the virtual machine model 26, i.e., the virtually replicated machine, moves within the virtual image model 30, i.e., the virtually replicated environment 16. The simulation models how the actual machine 6 is likely to move and behave in real time within the actual environment 16 in response to the control commands 18.

[0060] Device 4 also includes a parameter input 32. Device 4 receives real machine parameters 34 via this input, which contain information about machine 6 and its current state. The machine parameters 34 are the current speed, deceleration, acceleration, and steering angle of the vehicle or machine 6. Device 4 also includes a machine correction module 36. This module processes the machine parameters 34 and performs a standard comparison with the machine model 26 and, if necessary, corrects it to replicate or simulate the real machine 6 as accurately as possible in the virtual machine model 26. A dead-time element 38 serves to compensate for the time difference between the real machine 6, or the acquisition time of the machine parameters 34, and the simulation time of the machine model 26.In this case, the corresponding dead time is composed of the dead time TTU of the transmission of the control command 18 from the control room 8 to the machine 6 and the dead time TTY of the transmission of the machine parameter 34 from the vehicle 6 or the environment 16 to the device 4, each via the data network 20.

[0061] Machine 6 is assigned an imaging module 40 of machine system 2, in this case a camera system mounted on machine 6 or the vehicle, which captures images of the surroundings 16 from the perspective of machine 6. The resulting images are referred to here as machine image 42. Machine image 42 therefore depicts the real surroundings 16 of the real machine 6.

[0062] Device 4 has an image input 44, through which the machine images 42 are received from the imaging module 40 via the data network 20 (transmission or latency in the form of dead time TTV). The machine images 42 are transmitted to the image monitoring module 28 and thus contribute real image content to the simulation of the image model 30. The image model 30 therefore also contains real image components. This integration is achieved by appropriately correcting the dead times (not shown) and using suitable image processing.

[0063] Device 4 also includes an image correction module 46. This module is configured to subsequently correct the virtual image model 30 based on the (delayed arrival) machine images 42. Here, too, a dead-time element 38 is used in the usual manner, which takes into account the latency times TTU and TTV. As in the machine correction module 36, the correction of the image model 30 is carried out here in the usual manner by dead-time-adjusted comparison, determination of deviation, and error minimization between image model 30 and the real machine image 42.

[0064] Device 4 also includes an image machine correction module 48. This module serves to use the machine image 42 (in addition to the machine parameter 34) for checking and, if necessary, correcting the machine model 26. Thus, the machine model 26 is also updated based on the machine images 42 to reflect reality.

[0065] Facility 4 also contains environmental information 50 for machine 6, as well as an information input 52 for feeding environmental information 50 into facility 4. All of this is only symbolically indicated here. In this example, environmental information 50 is 3D map information of environment 16, here a 3D model of the mine / tunnel in which vehicle / machine 6 is moving. Furthermore, environmental information 50 contains the current position of other vehicles within the mine that are not shown, etc. This environmental information 50 is also used to simulate machine model 26 and image model 30 more realistically and with improved accuracy, for example, by including other vehicles in the simulation.

[0066] Fig. Figure 2 shows the temporal relationships of the processes in a time-based flowchart over time t. Fig. 1, in particular with regard to operator 10, machine 6 and imaging module 40.

[0067] The top line indicates to operator 10 that the display module 14, in the form of the monitor indicated by a grid of lines, displays a new, updated monitor image after a refresh rate TB of 50 ms (20 frames per second, fps). At time T1, an operator input (steering command) in the form of control command 18 is received by the HMI 12.

[0068] The control command 18 is then transmitted to the vehicle in the form of machine 6. Due to signal transmission times and processing times, the implementation of the command in the vehicle or machine 6 begins at time T2, i.e., after the dead time TTU. The implementation of the control command in machine 6 is shown in the second line of Fig. 2 is indicated by a jump function. At the same time, status variables in the form of machine parameters 34 are determined in the vehicle and sent back to the control room 8 as well as to the facility 4, where they arrive at time T3 (latency TTY).

[0069] The transmission of the control command 18 and feedback of the machine parameters 34 is indicated by arrows in Fig. 1 symbolizes.

[0070] Another control command 18 is transmitted to vehicle 6 at time T3 and arrives there at time T4. New machine parameters 34 are sent back from vehicle 6 to control room 10 and facility 4 at time T4 and arrive there at time T5.

[0071] The imaging system (imaging module 40) on the vehicle or machine 6 (front camera, multi-camera system) has to process (period T2 to T6) and transmit (T6 to T7) larger amounts of data, which means that current image data only arrives at the control room 8 and the facility 4 at time T7. Image processing in imaging module 40 therefore takes place from time T2 to time T6, and the transmission of the machine image 42 from time T6 to time T7.

[0072] The processes in imaging module 40 / provision of a real-world environment image are shown in the bottom line of Fig. 2 is indicated by a step function. The dead time TTY thus extends from time T2 to time T3, and the dead time TTV from time T2 to time T7.

[0073] If the real image data in the form of the machine image 42 were presented to the operator 10 at time T7, it would give the impression of delayed vehicle behavior (reaction to the steering command / control command 18 at time T1). Consequently, the operator 10 would tend to overcorrect steering movements or exit a curve too late. To simplify the operator 10's work, the inventive method or device 4 presents the vehicle reaction virtually instantaneously at a frame rate adapted to human visual perception, in this case 20 frames per second. Specifically, the device 4 is configured to display the images (from the image model 30 on the display module 14) instantaneously, i.e., in real time, by displaying them within a delay time ZV, which here corresponds to a predefinable maximum time ZM, which in this case corresponds to the frame repetition time TB.

[0074] Thus, the following applies: The very next monitor image after issuing the steering command, i.e., control command 18, at time T1, displays the image model 30, simulated in real time, on the display module 14. The operator 10 is therefore shown the simulated reaction of the machine 6 in the environment 16 in real time, in which a representation of the virtual machine model 26 is displayed in the virtual image model 30 (symbolically represented in the figure above by a jump function). In other words, a virtually delay-free transmission of the real machine image 42 is simulated.

Claims

[1] Device (4) for monitoring a real machine (6) located in a real environment (16) which can be controlled by a human operator (10) using a real control command (18), - with a control input (22) for inputting the control command (18), - with a machine observer module (24) which is designed to simulate a virtual machine model (26) of the machine (6) in real time based on the input control command (18), - with an image observer module (28) which is designed to simulate a virtual image model (30) of the environment (16) of the machine (6) in real time using the virtual machine model (26) simulated in real time and to output the virtual image model (30) simulated in real time to a display module (14), in particular the device (4), for display by means of the display module (14) in real time. [2] Device (4) according to claim 1, characterized bythat the device (4) is set up to simulate the machine model (26) by means of the machine observer module (24) and the image model (30) by means of the image observer module (28) in real time within a delay time (ZV) from the input of the control command (18) and to display them by means of the display module (14), wherein the delay time (ZV) is at most a predeterminable maximum time (ZM). [3] Device (4) according to claim 2, characterized by that the maximum time (ZM) is a single or multiple of a refresh time (TB) of the display module (14). [4] Device (4) according to one of the preceding claims, characterized by that the institution (4) - a parameter input (32) for receiving a real machine parameter (34), - and a machine correction module (36) which is configured to correct the machine model (26) based on the machine parameter (34). [5] Device (4) according to one of the preceding claims, characterized by , that - the machine (6) is assigned an imaging module (40) which is designed to generate a real machine image (42) of the environment (16) of the machine (6), - the device (4) contains an image input (44) for receiving the machine image (42), and - the image observer module (28) is configured to simulate the virtual image model (30) of the environment (16) of the machine (6) in real time, also using the machine image (42). [6] Device (4) according to claim 5, characterized by that the device (4) contains an image correction module (46) which is designed to correct the image model (30) based on the machine image (42). [7] Device (4) according to one of the preceding claims, characterized bythat the device (4) contains an image machine correction module (48) which is designed to correct the machine model (26) based on the machine image (42) and optionally also the machine parameter (34). [8] Device (4) according to one of the preceding claims, characterized by that the institution (4) - contains environmental information (50) of the machine (6) and / or an information input (52) for the environmental information (50), - and is configured to also simulate the machine model (26) and / or the image model (30) based on the environmental information (50). [9] Machine system (2), with the device (4) according to one of the preceding claims, and with the machine (6). [10] Machine system (2) according to claim 9, characterized bythat the machine system (2) comprises a data network (20) via which the machine (6) and the device (4) are communicatively connected and which, at least temporarily, is not capable of providing a real image of the environment (16) in real time. [11] Method for monitoring a real machine (6) located in a real environment (16) which can be controlled by a human operator (10) using a real control command (18), by means of a device (4) with the following steps: - receiving the control command (18) by means of a control input (22) of the device (4); - Simulating a virtual machine model (26) of the machine (6) based on the received control command (18) in real time by means of a machine observer module (24) of the device (4); - simulating a virtual image model (30) of the environment (16) of the machine (6) based on the virtual machine model (26) simulated in real time by means of an image observer module (28) of the device; and - Outputting the virtual image model (30) simulated in real time to a display module (14), in particular the device (4), for display by means of the display module (14) in real time. [12] Computer program comprising instructions which cause the device (4) according to one of claims 1 to 8 and / or the machine system (2) according to claim 9 or 10 to carry out the method steps according to claim 11. [13] A computer-readable medium on which the computer program according to claim 12 is stored.

Images