Method for remote control driving of a motor vehicle including a remote control operator, computer program product, and remote control driving system

Abstract

One aspect of the invention relates to a method for remotely controlling a motor vehicle (2), characterized in that a transport journey of at least one passenger (23) in the motor vehicle (2) is performed by means of the motor vehicle (2); during the transport journey, the driving of the motor vehicle (2) is performed at least in stages by an external remote operator (10) from an operator position (3) in a remotely controlled manner, wherein during the remotely controlled driving, environmental information of the motor vehicle (2) captured by at least one optical capture unit (17, 18, 19, 20) of the motor vehicle (2) is transmitted to the operator position (3), The invention also relates to a computer program product and a remote control driving system (1) for a remote control operator (10), wherein during remote control driving, the viewing direction (24) of the remote control operator (10) on the display unit (4, 5, 6, 7) is determined, and the image area (26) of the image (27) displayed on the display unit (4 to 7) at which the remote control operator (10) is looking is captured, wherein the image area (26) at which the remote control operator (10) is looking on his display unit (4, 5, 6, 7) is visually marked on a display unit (28) in the motor vehicle (2).

Description

Technical Field

[0001] One aspect of the invention relates to a method for remote control driving of a motor vehicle. Another aspect of the invention relates to a computer program product. Yet another aspect of the invention relates to an electronic remote control driving system for a motor vehicle. Background Art

[0002] Remote control driving of motor vehicles is known. Solutions are known in which the vehicle user leaves the vehicle and uses a manually operated remote control to perform vehicle maneuvers, particularly for parking. However, this is only possible for short, brief maneuvers, such as parking a vehicle. Furthermore, the user performing the remote control must be in the immediate vicinity of the vehicle.

[0003] Remote control driving of motor vehicles is also known. In these situations, the operator is located remotely from the vehicle, particularly to enable more comprehensive and extended vehicle control, even over long distances and in regular traffic. These remote control driving maneuvers are relatively complex, requiring a high level of focus to safely execute. Summary of the Invention

[0004] The object of the present invention is to provide a method in which a specific remote-controlled driving of a motor vehicle can be performed more safely. It is also an object to provide a computer program product and an electronic remote-controlled driving system.

[0005] One aspect of the present invention relates to a method for remote control driving of a motor vehicle. A journey for passengers in the motor vehicle is carried out via the motor vehicle. During the journey or journey, the motor vehicle is driven at least partially by an external remote operator from an operator's location in a remote control manner. This external operator may also be referred to as the remote operator. During the remote control driving, environmental information about the motor vehicle captured by at least one optical capture unit of the motor vehicle is transmitted to the operator's location. This transmitted environmental information is displayed to the operator on at least one display unit at the operator's location. During the remote control driving, the remote operator's viewing direction of the display unit is determined. The image area of ​​the image displayed on the display unit that the remote operator is looking at is captured. The image area on the display unit that the remote operator is looking at is marked on the display unit in the motor vehicle. This configuration facilitates passengers in the motor vehicle to identify where the operator is looking from the operator's location, whether the operator's location is outside and away from the motor vehicle, or which image area the operator is looking at. Thus, passengers in the motor vehicle can determine whether the operator is paying attention to the remote control driving of the vehicle. Thus, even passengers in a motor vehicle can recognize whether the remote operator is performing remote-controlled driving of the motor vehicle in an attentive manner. In particular, this can also detect whether the remote operator is directing their gaze towards at least one image area in the image that is relevant for driving the motor vehicle. Thus, in particular, passengers in a motor vehicle can recognize whether the remote operator is focusing on a critical sub-area of ​​the surrounding area. In particular, this focus on such image areas is relevant for the continued safe driving of the motor vehicle. Thus, in particular, passengers in a motor vehicle can also assess whether the remote operator is safely driving the motor vehicle remotely. Therefore, during such remote-controlled driving of a motor vehicle, confidence-building measures for passengers in the motor vehicle are also facilitated.

[0006] In particular, as one aspect of the present invention, a method for observing a remote control operator is provided. During this observation of the remote control operator, their viewing direction is captured and evaluated at their operator position, which is located outside and away from the motor vehicle. In this regard, the viewing direction of an image displayed on a display unit at the operator position is determined. In particular, this viewing direction is captured by an eye-tracking device located at the operator position. In particular, the intersection of the operator's viewing direction vector with the image plane is determined. Based on this intersection, the image region the operator is focusing on can also be determined. In this method for observing a remote control operator, at least one subregion of the image displayed on the display unit at the operator position is also displayed on a display unit in the motor vehicle. In the image on the motor vehicle's display unit, the image region currently being focused on by the remote control operator is visually marked. This is performed, in particular, during a passenger transportation journey in a motor vehicle. In particular, during the observation of the remote control operator's viewing direction, the image region observed by the viewing direction is displayed on the motor vehicle's display unit. In particular, this is achieved in an image which at least partially displays the viewing image area of ​​the operator.

[0007] In summary, the present invention facilitates safe remotely operated driving of a motor vehicle.

[0008] In the case of remote-controlled driving of a motor vehicle, the operator's location is typically within a building. This location is typically located far from the motor vehicle, allowing for remote control. Therefore, the remote operator cannot directly see the vehicle. This can range from hundreds or even thousands of kilometers. In particular, at least one display unit is located at each operator's location. These display units are preferably multiple, independent units. It is preferably envisioned that each of these display units displays a separate image captured by one or more camera units of the motor vehicle. For example, it is conceivable that an image captured by a first optical capture unit of the motor vehicle be displayed on a display unit. An image captured by a second optical capture unit of the motor vehicle can be displayed on a second display unit at the operator's location. An image captured by a third optical capture unit of the motor vehicle can be displayed on a third display unit at the operator's location. For example, it is conceivable that an image captured by a fourth optical capture unit of the motor vehicle be displayed on a fourth display unit at the operator's location. In particular, the optical capture units can be arranged on the motor vehicle so as to create a quasi-continuous detection range circumferentially around the motor vehicle. This allows the capture and display of the surrounding area surrounding the motor vehicle.

[0009] In an advantageous embodiment, the remote-controlled driving system includes a wireless communication link in addition to the operator's location. For example, a wireless communication link to a computing unit, particularly a computing center, can be provided. Corresponding data can be stored or exchanged via this link. For example, a connection to the internet can also be provided. Similarly, a wireless communication link between the motor vehicle and the internet can also be implemented. This could be via a mobile radio channel, for example. In particular, this could be implemented based on the 5G mobile radio standard.

[0010] In particular, a steering device is provided at the operator's station. This steering device, which can be, for example, a steering wheel, is actuated by the remote operator. This allows the remote operator to experience the same sensation as if they were actually operating the corresponding functional components in the motor vehicle, thus performing a near-realistic remote control operation.

[0011] In particular, it is also envisaged that an acoustic communication device is arranged at the operator's position. Thus, acoustic communication can be carried out between the remote control operator and the passengers in the motor vehicle.

[0012] It is also conceivable to provide the motor vehicle with another optical capture unit that is not a camera. In particular, as described above, the at least one image can be captured by at least one camera. Furthermore, the motor vehicle can include at least one laser scanner. This laser scanner, which can be, for example, a lidar sensor, allows for precise determination of the position of objects in the surrounding area. In particular, this makes it possible to determine the position of objects in the surrounding area more precisely than with a camera. However, other sensors can also be used instead of a laser scanner to determine the position of objects in the surrounding area of ​​the motor vehicle.

[0013] In particular, based on the environmental information obtained by the laser scanner, it is possible to more accurately determine the image area in which the operator is gazing at the image on their display unit. In this way, corresponding areas of the capture unit's field of view, or detection range, can be assigned to corresponding image areas of the image on the display unit. Similarly, the environmental data provided by the laser scanner can also be assigned to corresponding image areas of the image on the display unit. This means that if the remote operator is gazing at a certain area or location on the display unit, it is easy to determine which actual location in the vehicle's surroundings corresponds to the location the remote operator is gazing at. Thus, the output of the laser scanner, i.e., the environmental data, can be projected onto the field of view, or detection range, of the motor vehicle's optical capture unit, configured as a camera, in order to determine which pixels of the image on the display unit correspond to which parts of real-world objects in the motor vehicle's surroundings. This allows for even more precise determination of which image area the remote operator is actually gazing at.

[0014] Preferably, an image is at least partially displayed on a display unit in the motor vehicle, the image being displayed on the display unit at the operator's location. In particular, the image is displayed on the display unit in the motor vehicle, including the image area on which the operator is looking at their display unit. Thus, at any time during the transport journey, passengers in the motor vehicle can identify the operator's current primary visual focus.

[0015] Preferably, the image displayed on the display unit at the operator's location is temporally synchronized with the image displayed on the display unit in the motor vehicle. This design facilitates achieving a particularly realistic and quasi-identical display on both display units. In particular, passengers in the motor vehicle can thus identify the operator's current viewing direction from the operator's location. Thus, it is immediately clear to the passengers where the operator is looking. In a preferred embodiment, an image of the surrounding area captured by the motor vehicle's optical capture unit is displayed on the display unit at the operator's location. This is another highly advantageous embodiment for safe remote control operation of the motor vehicle. The operator can thus identify the vehicle's surrounding area in real time through a corresponding image representation on their display unit. This allows remote control operation of the motor vehicle as if the remote operator were sitting directly in the motor vehicle. This enables highly realistic and safe remote control operation of the motor vehicle.

[0016] Preferably, an optical capture unit captures images of the surrounding area as a video recording, and displays the video recording on a display unit at the operator's location. Basically, the formulas for capturing and / or displaying images allow for both static frame representation and video representation. Thus, video representation enables a dynamic representation process. Thus, the perceived and dynamically changing surrounding area during the movement of the motor vehicle can be displayed to the remote operator and the vehicle's passengers on the corresponding display unit. This promotes safe and realistic remote-controlled driving. This also facilitates the dynamic representation and / or changes in the remote operator's viewing direction, visually marked in real time on the image on the vehicle's display unit. This also makes it possible to present the remote operator's observations to the vehicle's passengers in real time. Conversely, the visual markings in the image on the vehicle's display unit provide a more or less indirect representation of the remote operator's surroundings at the operator's location.

[0017] Thus, a situation is quasi-optically presented to a passenger in a motor vehicle, in which he or she can precisely identify those image areas at which the remote operator is looking from the perspective of the remote operator's image. Thus, the passenger in the motor vehicle is optically shown how the remote driver of the vehicle perceives the surrounding area.

[0018] In an advantageous embodiment, it is contemplated that during remote-controlled driving of a motor vehicle, the remote operator's viewing direction is permanently captured. In particular, changes in viewing direction are also captured dynamically. In particular, such changes in the remote operator's viewing direction and / or static viewing direction are permanently displayed on a display unit in the motor vehicle. In particular, this is achieved in an image displayed on the display unit in the motor vehicle.

[0019] It is conceivable that the marking of the remote operator's current viewing direction and / or the change of the viewing direction on a display unit in the motor vehicle can be implemented in a dynamically variable manner. In particular, it is conceivable that this dynamic display is synchronized with the actual behavior of the remote operator at the operator's location. In particular, the marking of the current viewing direction and / or the change of the viewing direction can be displayed in real time on the image of the display unit in the motor vehicle. This also makes it easy to immediately identify the actual viewing situation of the remote operator in the motor vehicle. This also ensures high safety for passengers in the motor vehicle. They thus know, in particular constantly, the actual viewing pattern and behavior of the remote operator at the currently indicated time.

[0020] Preferably, it is contemplated that, in the image displayed on the display unit in the motor vehicle, the image area currently being viewed by the operator on the display unit is indicated by color. Additionally or alternatively, a region boundary outline can also be visually displayed. Passengers in the motor vehicle can easily discern the color signal. This allows for rapid identification of the remote operator's attention to surrounding areas relevant to safe remote control operation of the motor vehicle. For example, the color signal can be implemented to create the effect of representing an image area as a surface in a signal color. In particular, it is contemplated that this color signal can be opaque or transparent. This means that the surrounding area represented by the image area in the image remains recognizable through the color signal transmission. Consequently, the entire displayed image is not undesirably disrupted or rendered unintelligible by the color dots. Instead, the entire surrounding area represented by the image remains recognizable, with the color signal superimposed on it. This allows for a clear visual identification of the surface area currently being viewed by the remote operator. In additional or alternative embodiments, by displaying only a region boundary outline of the viewed image area, the image area currently being viewed by the remote operator can be easily identified in a manner that is equally easy to understand and does not obscure other image information. The boundary outline can be, for example, a solid or dashed line. In particular, if the color signal is unsuitable due to environmental influences, such as the corresponding brightness, then only the area boundary outline can be displayed. Similarly, if the area of ​​the environment currently being viewed by the remote operator in the image, and subsequently displayed using a color signal on the display unit in the motor vehicle, is so important and relevant that such a particularly opaque color overlay is undesirable, this is also possible. Therefore, the remote control driving system can also determine how the corresponding visual marking in the image on the display unit in the motor vehicle should be implemented, based on specific criteria. It is also possible that the color signal and / or area boundary outline changes depending on traffic conditions and / or objects detected in the motor vehicle's surrounding area, the remote operator's previous viewing direction, the duration of the remote operator's gaze in a particular viewing direction, and / or the duration of the remote operator's current gaze in a particular viewing direction. For example, this could involve a color change and / or a dynamic flashing of the color signal. Similarly, the area boundary outline can also be subjected to the same conditions. Consequently, more than just static optical marking of the remote operator's viewing direction is performed. Conversely, the remote operator's past and / or current behavior and / or the current and / or future situation regarding the viewing direction in the vehicle's surrounding area can also be taken into account. This facilitates visually marking the remote operator's viewing direction on the vehicle's display unit, which is better adapted to the situation. This allows for different states of attention and / or different safety criticalities when driving a motor vehicle. In particular, this makes it possible to more clearly inform the vehicle's occupants of their level of attentiveness based on these aforementioned evaluation criteria.In particular, passengers can use this to better determine how to assess the remote operator's attention level. This also allows for a better assessment of passenger behavior. This typically involves corresponding voice communication with the remote operator, particularly regarding his / her viewing direction and / or his / her attention level.

[0021] Advantageously, in this respect, the passenger can also better recognize and / or assess whether the remote operator recognizes and / or correctly assesses the information in the surrounding area in order to perform safe remotely operated driving of the motor vehicle.

[0022] In particular, the passenger can also recognize if the remote operator fails to recognize information in the surrounding area, in particular if no information in the surrounding area is provided, because it could not be captured or only insufficiently captured by the optical capture unit of the motor vehicle.

[0023] In an advantageous embodiment, it is contemplated that if the operator's standard of attention is exceeded, the passenger transmits action information to the operator based on the viewing direction indicated by the operator on a display unit in the motor vehicle. This could be the case, for example, in the example described above. In particular, the passenger may request or request a change in the remote-controlled driving of the motor vehicle, also in relation to an assessment of how safe driving of the motor vehicle is being performed.

[0024] In particular, such visual markings on the image of the motor vehicle's display unit can be used to indicate, in particular by the passenger, whether the remote operator, for the safe driving of the motor vehicle, is paying attention to the relevant traffic areas. Such traffic areas can be, for example, crosswalks, sidewalks, bicycle lanes, etc. Confused intersections or unclear entrances and exits can also be considered such traffic areas.

[0025] In an advantageous embodiment, it is contemplated that a voice signal is transmitted from the passenger to the remote operator as action information. This can be accomplished via a communication link, in particular a wireless communication link. The action information can include information regarding a traffic zone that the remote operator is not currently focused on. However, the action information can also, in addition or alternatively, be instructions from the passenger regarding the manner in which the remote operator intends to conduct the remote-controlled driving of the motor vehicle.

[0026] It is conceivable that the safety category of the traffic area can be predefined by the passenger. This can occur, in particular, before the journey begins. This can be achieved, in particular, through a passenger-specific user profile. This information can also be stored using the remote control driving system. Furthermore, the passenger's traffic situation classification can be predefined, in particular, before the journey begins. This information can also be stored in an electronic user profile. This electronic user profile can be transmitted to the operator before the remote control driving begins. The remote control operator can then perform the remote control driving of their motor vehicle based on this user profile. This can also enable correspondingly adapted remote control driving. In particular, this information from the passenger's user profile can be displayed on a display unit at the operator's station. Additionally or alternatively, a visual image can be displayed on the operator's display unit at the operator's station. This can, in particular, remind the operator of important matters for the passenger while driving the vehicle. This can, in particular, enhance the passenger's sense of security during the remote control driving of the vehicle.

[0027] In particular, if the passenger knows the route of the transport journey, and in particular if he knows key traffic points, this also increases the awareness of the remote operator and allows him to be better prepared for the remote-controlled driving of the vehicle. In particular, the awareness of the remote operator is increased in this regard for his remote-controlled transport journey.

[0028] It is also conceivable to divide the surrounding area into multiple zones, each assigned to a safety category, based on the surrounding area captured by the optical capture unit and / or the surrounding area not or not yet captured by the optical capture unit but recognized by the passenger. Based on this classification, the optically marked image areas are evaluated, and if the marked image areas deviate from the safety category of a zone greater than a safety threshold, action information is transmitted to the remote operator. This design not only makes remote-controlled driving safer, but also improves the subjective sense of safety of the passengers in the motor vehicle. This makes it possible to operate a motor vehicle suitable for a single passenger.

[0029] Another aspect of the invention relates to a computer program product having instructions which, if executed by a computer, cause the computer to perform steps according to the above-named method or advantageous embodiments thereof.

[0030] Another aspect of the present invention relates to a remotely operated driving system. This system includes an operator station located outside and remote from the motor vehicle to be driven remotely. Furthermore, it includes at least one optical capture unit located on the motor vehicle. Furthermore, the remotely operated driving system includes a display unit in the motor vehicle and a display unit at the operator station. The remotely operated driving system includes at least one control and / or computing unit configured to execute a method according to the aforementioned aspect or advantageous embodiments thereof. In particular, the remotely operated driving system includes a computer program product as described above. In this regard, the control unit may be a computer.

[0031] Preferably, the teleoperated driving system includes at least one eye-tracking device, which is arranged at the operator's location. This can detect the operator's viewing direction. Additionally or alternatively, the teleoperated driving system can include at least one laser scanner, which is arranged on the motor vehicle and can detect the position of objects in the motor vehicle's surrounding area. In this context, the objects can be static or dynamic. These can be other traffic participants, such as pedestrians, cyclists, motorcyclists, or other vehicles. However, they can also be boundary markers, buildings, trees, bushes, parking lots, sidewalks, cycle paths, crosswalks, or any other type of static object corresponding to a traffic sign. This list of potential objects in the surrounding area should not be understood as a definitive list.

[0032] Specifically, the remote-controlled driving system further includes at least one microphone located on the motor vehicle. The microphone captures noises in the vehicle's surroundings and / or interior. These noises are transmitted to a unit located at the operator's location, where they are output as acoustic signals to the operator. This improves the operator's perception and understanding of the images displayed on the display unit at the operator's location, making them more realistic. Consequently, the operator can better understand and assess the traffic situation represented by the images.

[0033] Another aspect of the invention relates to a motor vehicle comprising a display unit and at least one communication unit, by means of which a communication possibility with an operator's location is provided.

[0034] Further features of the invention are apparent from the claims, the drawings and the description of the drawings. The features and feature combinations mentioned in the above description and the features and feature combinations mentioned in the following description of the drawings and / or shown individually in the drawings can be used not only in the respectively specified combination, but also in other combinations without departing from the scope of the invention. Therefore, embodiments are also considered to be encompassed and disclosed by the invention, which are not explicitly shown and explained in the drawings, but are produced and generated by separate feature combinations from the explained embodiments. Embodiments and feature combinations are also considered to be disclosed and therefore do not include all features of the originally formulated independent claims. Furthermore, embodiments and feature combinations should be considered to be disclosed, in particular by the embodiments explained above, which extend beyond or deviate from the feature combinations explained in the back-references of the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] An embodiment of the present invention is explained in more detail below based on schematic diagrams.

[0036] These figures show:

[0037] Figure 1 is a schematic diagram of an embodiment of a remotely operated driving system;

[0038] Figure 2 is based on Figure 1 A schematic diagram of the operator position of the remotely operated pilot system;

[0039] Figure 3 is a representation of an image displayed on a display unit of a motor vehicle being driven in a remotely controlled manner; and

[0040] Figure 4 is a schematic flow chart of an embodiment of a method according to the present invention.

[0041] In the figures, identical elements or elements having the same function are provided with the same reference numerals. DETAILED DESCRIPTION

[0042] exist Figure 1 In the schematic diagram of , an electronic remote-controlled driving system 1 is shown. With such a remote-controlled driving system 1, a motor vehicle 2 can be driven remotely, i.e. in a remote-controlled manner. The motor vehicle 2 is intended for transporting people. The remote-controlled driving system 1 has an operator position 3. This is located in particular in a building. The operator position 3 comprises at least one display unit. In this embodiment, a plurality of display units are provided, in particular a display unit 4, a further display unit 5, a further display unit 6. In particular, at least one fourth display unit 7 is arranged at the operator position 3. The display units 4 to 7 are in particular screens. Furthermore, the operator position 3 comprises a steering unit, in particular a steering wheel 8. Furthermore, an acoustic communication unit 9 is arranged at the operator position 3. In Figure 1 An operator (also referred to as remote operator 10 and a person), which is shown only symbolically in FIG, is present at operator position 3 . Remote operator 10 performs remote-controlled driving of motor vehicle 2 .

[0043] In an advantageous embodiment, operator location 3 can communicate wirelessly with a computing unit 12, in particular a server, via a wireless communication link 11. This computing unit 12 can be connected, for example, to the internet 14 via a communication link 13. Motor vehicle 2 can communicate wirelessly with internet 14 via another communication link 15. In particular, communication can be performed using a mobile radio standard, in particular the 5G standard. A direct communication link between motor vehicle 2 and operator location 3 can also be provided, in particular using the 5G standard.

[0044] The motor vehicle 2 is in particular a fully autonomous vehicle. It has in particular an autonomy level 5. It is therefore a driverless motor vehicle 2 .

[0045] Motor vehicle 2 includes at least one capture unit, by means of which an environment area 16 of motor vehicle 2 can be captured. In particular, motor vehicle 2 includes an optical capture unit for this purpose. In particular, a camera 17 can be provided as an optical capture unit, which is sensitive in the human-visible spectral range. This camera is configured to capture an environment subarea located in front of motor vehicle 2. Furthermore, motor vehicle 2 can include additional optical capture units 18, 19, and 20. These can also be cameras. The detection ranges of optical capture units 18 and 19 are arranged on opposite side areas of motor vehicle 2 and are oriented for capturing lateral environment subareas of environment area 16. Optical capture unit 20 is oriented rearwards to capture an environment subarea of ​​environment area 16 behind motor vehicle 2.

[0046] In an advantageous embodiment, the motor vehicle 2 can additionally comprise at least one laser scanner 21 . This detects a sub-area of ​​the surroundings, in particular in front of the motor vehicle 2 .

[0047] In particular, the motor vehicle 2 also comprises a control unit 22 .

[0048] Symbolically, a passenger 23 is located in a motor vehicle 2. Passenger 23 wishes to undertake a transport journey in motor vehicle 2, or actually undertakes a transport journey in motor vehicle 2. During the transport journey, the driving of motor vehicle 2 is at least partially, and particularly entirely, performed by an external remote operator 10 from an operator station 3 in a teleoperated manner. During this teleoperated driving, environmental information of an area 16 surrounding motor vehicle 2 is captured by at least one optical capture unit 17 to 20. This captured information, particularly images, particularly videos, is transmitted to operator station 3. At operator station 3, these images are displayed on at least one display unit 4 to 7. It is conceivable, for example, that the images captured by optical capture unit 17 be transmitted to operator station 3 and displayed there on centrally located display unit 5. It is conceivable that the images captured by optical capture unit 18 be transmitted to operator station 3 and displayed on right-hand display unit 6. It is conceivable that the images captured by optical capture unit 20 be displayed on display unit 7, which is arranged in the center and below display unit 5. In particular, it is envisaged that the image transmitted by the optical capture unit 19 is transmitted to the operator position 3 and displayed there on the display unit 4 arranged on the left.

[0049] During teleoperated driving of the motor vehicle 2, the viewing direction 24 of the teleoperator 10 is captured ( Figure 2 For this purpose, an eye tracking device 25 ( Figure 1 ). In this case, it is detected at which image region 26 of the image 27 displayed on the display unit 4 and / or 5 and / or 6 and / or 7 the remote operator 10 is directed. In particular, it is also determined at which point in the image the vector of the viewing direction 24 intersects or touches the image plane.

[0050] In particular, based on the information of the at least one laser scanner 21, the positions of real objects in the environment area 16 can be determined. In addition to this information and knowledge, whose image representation is implemented on the display unit 5, it is also possible to determine which image area or which object the teleoperator 10 is currently looking at.

[0051] The motor vehicle 2 comprises in particular at least one display unit 28 ( Figure 1). An image 29 is displayed on this display unit 28, which image in particular comprises a sub-area of ​​the image 27, at which the remote operator 10 is currently looking. This image 29 on the display unit 28 advantageously also shows at least the image area 26, at which the remote operator 10 is currently looking in the image 27. In particular, it is envisaged that this image area 26 is marked, in particular visually marked, on the image 29 of the display unit 28 in the motor vehicle 2, at which image area on its display unit, in this example the display unit 5, is currently looking. To this end, in this embodiment it is envisaged that this area 26 is marked or represented by a color. For example, a surface coloring can be implemented here that contrasts with other areas of the image 29. In particular, such a color signal is implemented transparently or opaquely. As a result, the actual surrounding information of this image area 26 can still be recognized by this coloring 30. Therefore, in this respect, for example Figure 3 In the image 29 , it can be seen that the remote operator 10 is looking at a specific area of ​​the sidewalk and / or bicycle path 31. This is marked in the image 29 by a color signal or coloring 30. Thus, the passenger 23 in the vehicle 2 can always and constantly recognize which sub-area of ​​the environment or which object the remote operator 10 is currently looking at.

[0052] In particular, it is contemplated that an image 27, which is displayed on display unit 5 at operator station 3, is at least partially displayed on display unit 28 in motor vehicle 2. In particular, image 27 displayed on display unit 5 at operator station 3 is temporally synchronized with image 29 displayed on display unit 28 in motor vehicle 2. In particular, it is contemplated that images captured by optical capture units 17 and / or 18 and / or 19 and / or 20 are displayed in real time on display units 4 and / or 5 and / or 6 and / or 7 at operator station 3. In particular, the captured images are recorded as a video and also displayed as a video on the corresponding display units 4 to 7, 28. In particular, during remote-controlled driving of motor vehicle 2, operator 10's viewing direction 24 is permanently captured. In particular, each viewing direction and changes in viewing direction can thus be captured continuously. Consequently, dynamic changes in viewing direction can also be captured without restriction. Furthermore, in this regard, a synchronized and real-time display of color signals 30 can be achieved, particularly in a dynamically variable manner. In addition to or instead of the surface color signal, the region boundary contour of the image region 26 can also be visually signaled. Depending on specific criteria, a change in color and / or a change in the region boundary contour can also be achieved. In this regard, dynamic changes can also be achieved.

[0053] It is conceivable that passenger 23 transmits action information to operator 10 based on viewing direction 24 marked by operator 10 on display unit 28. This is particularly true if the operator's 10 attentiveness level is insufficient. The action information can be a voice signal. This can be transmitted to operator position 3 via communication links 15, 13, and 11. Direct transmission from motor vehicle 2 to operator position 3 is also possible. In particular, by optically marking viewing direction 24 of remote operator 10 on the image in motor vehicle 2, it is also possible to detect whether remote operator 10 perceives a critical traffic situation and / or a specific traffic area as required, so that safe remote-controlled driving of motor vehicle 2 can be carried out.

[0054] It is also possible that the display on only one display unit 28 in the motor vehicle 2 can be changed dynamically. This is the case, for example, if the operator 10 in his operator position 3 changes his gaze from one display unit 4 to 7 to another display unit 4 to 7. In particular, if he changes his gaze, for example, according to Figure 2 , the operator 10 turns his gaze away from the image 27 on the display unit 5 and, for example, directs his gaze towards the image displayed on the display unit 4. This can also be detected by the eye tracking device 25. The image display on the display unit 28 of the motor vehicle 2 can then be changed synchronously and in real time. In particular, the image displayed on the display unit 5 of the operator position 3 is also at least partially displayed there. In particular, the image display on the display unit 28 is then changed so that the image area on the display unit 5 at which the operator 10 is currently looking is again at least partially displayed on the display unit 28. Thus, here too, an optical marking of the image area on the display unit 28 at which the operator 10 is currently looking can always be achieved.

[0055] Usually based on Figure 4 According to the simplified flowchart in FIG, it can be assumed that the scenario begins in step S1. In this regard, according to step S2, a check is performed to determine whether remote-controlled driving has begun. If this is not the case according to S3, a return is made to step S1. However, if remote-controlled driving has begun according to step S4, the gaze direction 24 of the operator 10 is determined in a further step S5. If no intersection of the gaze direction 24 with the image on the display unit of the operator position 3 is detected in this regard, a return is made to step S1 according to step S6. If a gaze direction 24 of the operator 10 intersecting with the image on the display devices 4 to 7 is detected, the process continues according to step S7 so that the observed image area on the image of the display unit in the motor vehicle 2 is visually marked, as shown in step S8.

Claims

1. A method for remotely controlling a motor vehicle (2), It is characterized by: - performing a transport journey of at least one passenger (23) in the motor vehicle (2) by means of the motor vehicle (2); - During the transport drive, the driving of the motor vehicle (2) is performed at least in stages by an external teleoperator (10) from an operator position (3) in a teleoperated manner, wherein: - During teleoperated driving, information about the surroundings of the motor vehicle (2) captured by at least one optical capture unit (17, 18, 19, 20) of the motor vehicle (2) is transmitted to the operator's location (3) and displayed there on at least one display unit (4, 5, 6, 7) for the teleoperator (10), wherein - during remote-controlled driving, determining the viewing direction (24) of the remote operator (10) on the display units (4, 5, 6, 7) and capturing which image area (26) of the image (27) displayed on the display units (4 to 7) the remote operator (10) is looking at, - transmitting the viewing direction (24) of the remote operator (10) to a display unit (28) in the motor vehicle (2) and marking the viewing direction (24) on the display unit (28), wherein - The image area (26) that the remote control operator (10) looks at on his display unit (4, 5, 6, 7) is visually marked on the display unit (28) in the motor vehicle (2).

2. The method according to claim 1, It is characterized by: On a display unit (28) in a motor vehicle (2), an image (29) is at least partially displayed, which image is displayed on a display unit (4, 5, 6, 7) at an operator position (3).

3. The method according to claim 1 or 2, It is characterized by: The image (27) displayed on the display unit (4, 5, 6, 7) of the operator's position (3) is displayed synchronously with the image (29) displayed on the display unit (28) in the motor vehicle (2).

4. The method according to any one of the preceding claims, It is characterized by: An image of an ambient area (16) captured by an optical capture unit (17 to 20) of a motor vehicle (2) is displayed in real time on a display unit (4 to 7) at an operator's position (3).

5. The method according to any one of the preceding claims, It is characterized by: The recording of images of the surrounding area (16) by the optical capture units (17 to 20) is performed as a video recording, and the display of the video recording is performed on the display units (4 to 7) at the operator position (3).

6. The method according to any one of the preceding claims, It is characterized by: During the teleoperated driving, the viewing direction (24) of the teleoperator (10) is permanently captured, in particular also changes in the viewing direction are captured dynamically.

7. The method according to claim 6, It is characterized by: The marking of the current viewing direction (24) and / or the changing of the viewing direction on a display unit (28) in a motor vehicle (2) is carried out in a dynamically variable manner, in particular synchronously in time, in particular in real time.

8. The method according to any one of the preceding claims, It is characterized by: In an image (29) displayed on a display unit (28) in a motor vehicle (2), the image region (26) currently being viewed by the remote operator (10) on his display unit (4 to 7) is indicated by color and / or by displaying a region boundary outline of the image region (26).

9. The method according to any one of the preceding claims, It is characterized by: The passenger (23) transmits action information to the remote operator (10) based on the viewing direction (24) marked by the remote operator (10) on a display unit (28) in the motor vehicle (2), in particular if the remote operator's (10) standard of attention is below.

10. The method according to claim 9, It is characterized by: As action information, a voice signal is transmitted via the communication link (15, 13, 11).

11. The method according to any one of the preceding claims, It is characterized by: Based on the environment area (16) captured by the optical capture units (17 to 20) and / or an environment sub-area passing through the environment area (16), which has not or has not yet been captured by the optical capture units (17 to 20) but has been recognized by the passenger (23), the environment area (16) is subdivided into a plurality of zones, which are assigned to respective safety categories, wherein, depending on the assigned safety category, the marked image area (26) is evaluated and, if the marked image area (26) deviates from a zone whose safety category is above a safety threshold, action information is transmitted to the remote operator (10).

12. The method according to any one of the preceding claims, It is characterized by: The classification of the traffic situation and / or traffic zone by the passenger (23) is predetermined, in particular before the start of the transport drive, in particular via an electronic passenger profile, wherein the classification is transmitted to the remote operator (10), in particular before the start of the remote-controlled drive.

13. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the steps of the method according to any one of the preceding claims.

14. A remotely operated driving system (1) for remotely operated driving of a motor vehicle (2), comprising an operator position (3) outside the motor vehicle (2), comprising at least one optical capture unit (17 to 20) of the motor vehicle (2), comprising a display unit (28) of the motor vehicle (2) and comprising at least one display unit (4 to 7) at the operator position (3), and comprising at least one control and / or computing unit (12, 22) configured to carry out the method according to any one of claims 1 to 12.

15. The remote-controlled driving system (1) according to claim 14, comprising an eye-tracking device (25) at the operator's position (3), by means of which the viewing direction (24) of the remote-controlled operator (10) can be detected, and / or comprising at least one laser scanner (21) of the motor vehicle (2), by means of which the position of objects in the surrounding area (16) can be detected.

Images