Remote control of vehicles

Abstract

The invention relates to remote-controlled driving of a vehicle, in particular to a method for remote-controlled driving of a vehicle (1), sensor data relating to the environment of the vehicle (1) are received, and a computing system (2) is used to determine the probability of an accident affecting the control of the vehicle (1) based on the sensor data. The computing system (2) is used to receive quality information relating to the predicted quality of a communication network (3) and to define at least one parameter for the remote control of the vehicle (1) based on the probability of the accident and based on the quality information.

Description

Technical Field

[0001] The invention relates to a method for remotely operated driving of a vehicle via a communication network, wherein the method comprises using a computing system of the vehicle to receive sensor data relating to the environment of the vehicle. The invention further relates to an electronic vehicle navigation system for a remotely operable vehicle and a computer program product. Background Art

[0002] Teleoperated driving can be seen as a potential key technology in the context of highly automated or fully automated vehicles. In the event that the autonomously driven vehicle is unable to interpret and resolve a certain situation, such as at a crash site or a construction site, these vehicles can use external instructions from a remotely operated control center. The vehicle is then driven remotely from the control center.

[0003] For remote driving, remote operation instructions must be transmitted from the control center to the vehicle via a communication network. As a result, if the communication link quality of the communication network is poor, the safety of remote operation driving may be compromised.

[0004] Document US 2018 / 0348750 A1 describes an approach for remote operation of an unmanned ground vehicle in which the quality of the communication channel is monitored and in the case of low quality (e.g., low available bandwidth or high latency), the amount of data transmitted from the vehicle via the network to the remote operation operator is reduced, for example, by constraining the transmitted video to a restricted area of ​​interest.

[0005] However, according to this approach, it cannot be ensured that the amount of data transferred is reduced enough to compensate for the degradation of network quality. In other words, this approach is only suitable for moderate degradation of communication quality. As a result, the risk of reduced security or confidentiality in the event of a significant degradation of network quality is still high.

[0006] Document US 2018 / 0136651 A1 discloses a remote operation system and method for trajectory modification of an autonomous vehicle. Sensor data is received from the modality of the autonomous vehicle, for example, data identifying static objects or dynamic objects can be derived from lidar data or camera data. The object is determined to have an impact on the planned path, and a subset of the trajectory is evaluated in response to the object. In addition, a service platform is described that implements redundant communication channels to maintain reliable communication with a fleet of autonomous vehicles. The rate of data exchanged between the autonomous vehicle and the platform can be changed based on the quality level of communication conducted through the channel. During bandwidth-constrained periods, data communications can be prioritized so that remote operation requests from autonomous vehicles are highly prioritized to ensure delivery. Summary of the invention

[0007] It is therefore an object of the present invention to provide an improved concept for remote-controlled driving of a vehicle which further increases the safety and confidentiality of the remote operation.

[0008] According to improved concepts, this object is achieved by the subject matter of the independent claims. Further embodiments and preferred embodiments are the subject matter of the dependent claims.

[0009] The improved concept is based on the idea of ​​using the predicted quality or predicted quality of service QoS of the communication network in combination with sensor data concerning the vehicle's environment in order to proactively define at least one parameter for remote operation of the vehicle, in particular before a possible actual safety-relevant accident may occur.

[0010] According to the improved concept, a method for remote-controlled driving of a vehicle via a communication network, in particular a computer-implemented method, is provided. The method comprises receiving sensor data relating to the environment of the vehicle using a computing system of the vehicle. The computing unit is used to: determine the probability of an accident affecting the control of the vehicle depending on the sensor data; receive quality information relating to the predicted quality of the communication network, in particular from the communication network and / or from a server computer; and define at least one parameter for the remote control of the vehicle depending on the probability of the accident and depending on the quality information.

[0011] Here and hereinafter, remote-operated driving of a vehicle can be understood such that a control unit of the vehicle (the computing system may include the control unit) automatically controls the vehicle based on remote-operation instructions, wherein the remote-operation instructions are received by the vehicle (in particular, the control unit) via the communication interface of the vehicle from a communication network, in particular, from a control center or a control computer via the communication network.

[0012] The remote operation instruction may be issued or generated, for example, by an operator operating the control computer.

[0013] The vehicle is particularly designed as a partially or fully automated vehicle. In particular, the vehicle can be designed as a highly automated vehicle of level 4 according to SAE J3015 classification dated June 2018 or as a fully automated vehicle of level 5 according to SAE J3015 classification.

[0014] The quality information relating to the predicted quality of the communication network can eg correspond to predicted quality of service PQoS information, in particular comprising a level or measure of the quality of service QoS of the communication network, eg an expected QoS or an expected QoS degradation of the communication network.

[0015] There are known methods for providers of communication networks to estimate or predict the QoS of the provider's communication network and to provide the QoS via the communication network itself.

[0016] Remote operation driving of a vehicle via a communication network can be understood as remote operation driving based on remote operation information transmitted from a control center to the vehicle via the communication network.

[0017] The communication network is designed in particular as a cellular network, in particular as a communication network in which the final link between the vehicle and the communication interface and the communication network is a wireless link.

[0018] The accident that affects the control of the vehicle may be an actual accident or an accident that has occurred, or may be an accident that will occur in the future with a certain probability.

[0019] The probability of an accident affecting control of the vehicle may depend on or be given by the probability that an accident occurs or in particular whether an accident has occurred or is already occurring, the probability that an accident actually affects control of the vehicle.

[0020] For example, an accident may correspond to another vehicle in the environment of the vehicle performing a braking maneuver. Thus, the probability of an accident affecting the control of the vehicle can depend on the probability that the other vehicle will actually perform a braking maneuver and / or the probability that the braking maneuver affects the control of the vehicle if it occurs.

[0021] An accident can for example also include the presence of a construction site. The probability that the accident affects the control of the vehicle can for example correspond to or be given by the probability that the location or other properties of the construction site actually affect the control of the vehicle.

[0022] The control of the vehicle is affected, for example, by an accident if control actions or control maneuvers for controlling the vehicle have to be initiated or adapted, in particular in order to maintain a level of safety or security.

[0023] Defining at least one parameter for remote operation of the vehicle includes defining constraints for operating the vehicle, such as a maximum velocity.

[0024] In particular, defining at least one parameter can, for example, comprise determining at least one parameter and optionally storing, for example temporarily, at least one parameter.

[0025] According to the method according to the improved concept, prevention of a scenario in which the QoS of the communication network is reduced and an accident that affects the control of the vehicle occurs at the same time is effectively taken by defining at least one parameter for remote operation. The accident may, for example, correspond to a worst-case scenario combined with a reduction in QoS. Thus, certain general conditions for remote-operated driving of the vehicle may be defined by defining at least one parameter.

[0026] In the case of remote operation which is necessary when the quality of the communication network is low, the vehicle can be controlled automatically as a function of at least one parameter, in particular in the absence of explicit instructions received via the communication network.

[0027] Thus, in the event of such a worst-case scenario, the vehicle does not have to come to a complete stop or substantially reduce speed to, for example, walking speed.

[0028] Thus, the comfort level for the user of the vehicle in a scenario as described can be increased, while at the same time the security of the confidentiality of the teleoperated driving remains at a high level.

[0029] According to several embodiments of the method for remote-operated driving of a vehicle according to the improved concept, during an autonomous driving mode of the vehicle, the vehicle is driven partially or fully automatically by a computing unit. The following steps can be performed during the autonomous driving mode: receiving sensor data; determining the probability of an accident affecting the control of the vehicle; receiving quality information; and defining at least one parameter for remote operation.

[0030] According to several embodiments, the control unit or computing system is configured to activate a remotely operated driving mode of the vehicle if a predefined scenario has occurred. During the remotely operated driving mode, the communication interface receives remote operation instructions from the communication network, and the control unit controls the vehicle automatically or remotely controlled based on the remote operation instructions.

[0031] Furthermore, during the remote operation mode, the following steps may be performed: receiving sensor data; determining a probability of an event affecting control of the vehicle; receiving quality information; and defining at least one parameter for the remote operation.

[0032] According to several implementations, a computing system is used to determine a respective probability of occurrence of each of one or more potential events and to select an incident from the one or more potential events depending on the respective probability of occurrence.

[0033] Events correspond to potential occurrences that may generally affect control of the vehicle.

[0034] An accident can then be selected, for example, as an event in the worst case scenario.

[0035] For example, an incident can be selected depending on the probability of the occurrence of the respective event and depending on the classification or category of the respective event and / or the impact of the event and / or other parameters of the event.

[0036] According to several implementations, a computing system is configured to receive first sensor data from a sensor system of a vehicle.

[0037] According to several embodiments, the first sensor data is generated by a sensor system of the vehicle.

[0038] The vehicle's sensor system may specifically include one or more environmental sensor systems (such as, cameras, lidar systems, radar systems and / or ultrasonic sensor systems) and / or one or more initial or motion sensors configured to determine the vehicle's motion parameters (such as, longitudinal velocity, longitudinal acceleration, lateral velocity, lateral acceleration, etc.).

[0039] According to several embodiments, the computing system is configured to receive second sensor data from a vehicle-to-vehicle (V2V) interface of the vehicle.

[0040] The second sensor data are generated, in particular, by one or more corresponding sensor systems of one or more further vehicles located in the environment of the vehicle.

[0041] By taking the second sensor data into account, collective sensing may be exploited to enhance the available sensor data and as a result a higher reliability of the method is achieved.

[0042] According to several embodiments, the computing system is configured to determine obstacle information related to potential obstacles in the environment involved in the accident based on the sensor data. The computing system is configured to define at least one parameter for the remote operation depending on the obstacle information.

[0043] Obstacle information can be determined, for example, based on one or more algorithms evaluating sensor data (e.g., including one or more computer vision algorithms, algorithms for data fusion and evaluation, etc.). In this way, it is possible to make a comprehensive assessment of the actual situation and a proper estimate of the worst-case scenario.

[0044] According to several embodiments, the obstacle information includes a distance of the obstacle from the vehicle and / or a speed of the obstacle.

[0045] The speed of the obstacle can, for example, comprise a relative speed with respect to the vehicle, or in other words, a speed difference between the speed of the obstacle and the speed of the vehicle.

[0046] The obstacle can be, for example, a fixed obstacle, which means that the obstacle does not or does not substantially move relative to the road on which the vehicle travels, such as a construction site.

[0047] The obstacle can also be a dynamic obstacle (such as another vehicle or a pedestrian), which means that the obstacle has a considerable speed relative to the road.

[0048] According to some embodiments, the obstacle information includes the category of the obstacle.

[0049] The category of an obstacle can correspond, for example, to the type of obstacle, in other words to semantic information assigned to the obstacle.

[0050] In particular, the computing unit can execute a classification algorithm based on the sensor data in order to assign a class to the obstacle.

[0051] Different types of objects may be more or less critical with respect to the control of the vehicle. Thus, the categories can be relevant indicators for an accident corresponding to a worst case scenario.

[0052] The category can, for example, relate to the movement, speed and / or size of the obstacle.

[0053] According to several embodiments, the computing system is configured to read a database to define at least one parameter for the remote operation, wherein the database relates the quality information and the obstacle information to the at least one parameter.

[0054] The database can be, for example, a database stored on a memory unit (eg, a computing unit) of the vehicle, or can be stored external to the vehicle.

[0055] The database can be implemented, for example, as a lookup table.

[0056] In particular, the database contains pre-calculated values ​​for at least one parameter taking into account the predicted quality and obstacle information.

[0057] In this way, low computational requirements for defining the at least one parameter and a high processing rate may be achieved.

[0058] According to several embodiments, the calculation system is used to define a maximum speed of the vehicle during operation depending on the probability of an accident and depending on the quality information.

[0059] As a result, security or privacy can be improved, while the maximum speed only has to be adapted to as great an extent as is actually required.

[0060] According to several embodiments, the predicted quality information comprises a predicted end-to-end delay of the communication network, a predicted data transmission rate of the communication network and / or a predicted packet error rate of the communication network or a combination of said quantities.

[0061] According to several embodiments, the computing system is used to identify that an accident has occurred or occurred depending on further sensor data, for example, the further sensor data can be received from another vehicle via a V2V interface or via a sensor system of the vehicle. The computing system is used to initiate remote operation of the vehicle via the communication network and control the vehicle at least partially automatically based on at least one parameter defined and depending on the actual quality of the communication network.

[0062] Initiating remote operation may, for example, include sending a query or request via a communications network to a control center to take over control of the vehicle.

[0063] In particular, remote operation is initiated by the computing system via the communication interface.

[0064] The vehicle is in particular controlled by the computing system based on the definition of the at least one parameter if the actual quality of the communication network is below a predefined threshold.

[0065] According to several embodiments, the computing system is configured to control the vehicle based at least partially automatically on the defined at least one parameter if the value for the actual quality of the communication network is less than a predefined threshold.

[0066] According to an improved concept, an electronic vehicle navigation system for a remotely operable vehicle is provided. The vehicle navigation system comprises a computing system configured to receive sensor data relating to an environment of the vehicle. The computing system is configured to determine a probability of an accident affecting control of the vehicle depending on the sensor data. The computing system is configured to receive quality information relating to a predicted quality of a communication network for remote operation of the vehicle. The computing system is configured to define at least one parameter for remote operation of the vehicle depending on the probability of the accident and depending on the quality information.

[0067] According to several embodiments, a vehicle navigation system includes a sensor system for a vehicle or a sensor system of the vehicle configured to generate first ones of the sensor data.

[0068] The sensor system is configured to be installed in or on a vehicle, or the sensor system is installed in or on a vehicle.

[0069] In particular, the sensor system is configured to sense an environment of the vehicle in order to generate first sensor data.

[0070] According to several embodiments, the sensor system includes at least one camera, at least one lidar system and / or at least one radar system.

[0071] According to several embodiments, the vehicle navigation system includes a vehicle-to-vehicle interface configured to receive second sensor data of the sensor data, in particular from one or more further vehicles in an environment of the vehicle.

[0072] Further embodiments of the electronic vehicle navigation system are directly inferred from the various embodiments of the method according to the improved concept, and vice versa. In particular, the vehicle navigation system according to the improved concept can be configured or programmed to implement the method according to the improved concept, or the vehicle navigation system implements or implements the method according to the improved concept.

[0073] According to an improved concept, a vehicle, in particular an at least partially autonomously drivable vehicle, is provided comprising an electronic vehicle navigation system according to the improved concept.

[0074] According to the improved concept, a computer program is provided comprising instructions which, when executed by a computer system, in particular an electronic vehicle navigation system according to the improved concept, such as a computing system of a vehicle navigation system, cause the computer system to implement the method according to the improved concept.

[0075] According to an improved concept, a computer readable storage medium storing a computer program according to the improved concept is provided.

[0076] The computer program and the computer-readable storage medium according to the improved concept can be regarded or referred to as a computer program product, respectively. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In the figure, it is shown that:

[0078] Figure 1 showing a schematic representation of a vehicle including an exemplary embodiment of an electronic vehicle navigation system according to the improved concept; and,

[0079] Figure 2 A flow chart showing an exemplary implementation of a method according to the improved concept. DETAILED DESCRIPTION

[0080] exist Figure 1 , a vehicle 1 is shown comprising an exemplary embodiment of an electronic vehicle navigation system 7 according to the improved concept.

[0081] The vehicle navigation system 7 includes a computing system 2 and a communication interface 8 for communicating via a cellular network 3 .

[0082] The vehicle navigation system 7 may further comprise a V2V interface 5 for communication between the vehicle 1 and further vehicles 1a, 1b in the environment of the vehicle. The vehicle navigation system 7 further comprises a sensor system 4, in particular one or more environment sensor systems.

[0083] The sensor system 4 may, for example, include a camera, a lidar system and / or a radar system.

[0084] The further vehicles 1 a , 1 b may, for example, each be equipped with a further sensor system 4 a , 4 b .

[0085] exist Figure 1 , obstacles 6 in the environment of the vehicle 1 and a control center 9 comprising a server computer 10 which can communicate via the network 3 , in particular with the communication interface 8 , are also schematically shown.

[0086] The control center 8 is used, for example, for remote operation of the vehicle 1. In particular, if the computing system 2 sends a request to the server computer 10 via the communication interface 8 and the network 3 to take over the control of the vehicle 1, the operator of the control center 9 can issue a remote operation instruction and send the remote operation instruction to the communication interface 8 via the server computer 10 and the network 3. Then, the computing system 2 can control the vehicle 1 depending on the remote operation instruction, so that the vehicle 1 can be remotely operated by the operator.

[0087] In the following, the functionality of the vehicle navigation system 7 is explained in more detail with regard to an exemplary embodiment of the method according to the improved concept.

[0088] exist Figure 2 A flow chart of such an exemplary implementation of a method according to the improved concept is depicted in .

[0089] In step S1 of the method, the computing system 2 receives quality information relating to the predicted quality of the network 3 (in particular, regarding a possible future QoS degradation on the end-to-end path from the vehicle 1 to the control center 9 via the uplink and back to the vehicle 1 via the downlink) via the network 3 and the communication interface 8. The quality information can be provided, for example, by a network operator of the network 3 or by the control center 9.

[0090] In step S2, the computing system receives sensor data relating to the environment of the vehicle 1. The sensor data may, for example, include first sensor data generated by the sensor system 4 and / or may contain second sensor data received via the V2V 5, which is, for example, implemented as a PC5 interface or communication module according to the standard IEEE802.11p. In other words, the second sensor data represents collective perception data.

[0091] In step S3, the computing system 2 determines the respective probabilities of occurrence of a plurality of potential events that may have an impact on the control of the vehicle 1. Events may include, for example, a braking maneuver of one of the other vehicles 1a, 1b, the presence of a construction site, pedestrians, intersections, the presence of traffic lights, etc.

[0092] In step S4 , the computing system 2 determines obstacle information about one or more potential obstacles in the environment, such as objects 6 or further vehicles 1 a , 1 b .

[0093] The obstacle information may, for example, include the distance between vehicle 1 and the obstacle, the corresponding velocity or velocity difference of the obstacle relative to vehicle 1, the maximum braking capacity of the obstacle in the case of other vehicles 1a, 1b and / or the probability of velocity adaptation of the obstacle in particular in the case of other vehicles 1a, 1b.

[0094] The probability of a speed adaptation may, for example, be regarded as low if no other vehicle is driving ahead of the respective further vehicle 1 a , 1 b .

[0095] Obstacle information can also include information about possible or potential trajectories of obstacles.

[0096] The obstacle information may be derived from the first sensor data or from the second sensor data, depending on the nature of the information.

[0097] Additionally, computing system 2 selects or identifies a worst-case incident from the one or more events.

[0098] In step S5 , the SC 2 defines at least one parameter for the remote operation of the vehicle 1 as a function of the probability of an accident in the worst case scenario and as a function of the quality information of the network 3 .

[0099] In other words, the computing system 2 may set one or more constraints for the teleoperation session, such as a maximum velocity for the vehicle 1. This can be done, for example, based on a database of pre-computed scenarios of future quality of service, distances, velocities, velocity differences, maximum braking capabilities, probabilities of velocity adaptation, etc., taking into account potential obstacles involved in a worst-case accident.

[0100] A degradation in QoS may for example result from congestion in the air interface, the network, connection problems in or at the control centre 9 , congestion in the control centre 9 and / or congestion in a slow server in the control centre 9 .

[0101] In step S6 , if the worst-case accident actually occurs and, at the same time, the QoS actually degrades, the vehicle 1 may be controlled by the computing system 2 depending on at least one parameter defined for the remote operation.

[0102] As described, according to the improved concept, situations in remotely operated driving where low network quality coincides with an incident affecting control of the vehicle can be handled to improve the safety of the remote operation and the comfort for the user of the remotely operated vehicle.

[0103] Reference Symbols

[0104] 1 Vehicle

[0105] 1a Vehicle

[0106] 1b Vehicle

[0107] 2 Computing System

[0108] 3 Network

[0109] 4 Sensor system

[0110] 4a Sensor system

[0111] 4b Sensor system

[0112] 5 Vehicle-to-Vehicle Interface

[0113] 6 Objects

[0114] 7 Vehicle Navigation System

[0115] 8 Communication Interface

[0116] 9 Control Center

[0117] 10 Server Computer

[0118] S1 Method steps

[0119] S2 Method Steps

[0120] S3 Method Steps

[0121] S4 Method steps

[0122] S5 Method Steps

[0123] S6 Methods and Steps

Claims

1. A method for remotely controlling a vehicle (1) via a communication network, wherein: The method comprises receiving, using a computing system (2) of the vehicle (1), sensor data relating to an environment of the vehicle (1), The computing system (2) is used for: - determining the probability of an accident affecting the control of the vehicle (1) based on the sensor data; - receiving quality information relating to a predicted quality of the communication network (3), wherein the quality information corresponds to predicted quality of service information; - defining at least one parameter for remote operation of the vehicle (1) depending on the probability of the accident and depending on the quality information, wherein defining the at least one parameter for the remote operation of the vehicle comprises defining constraints for operating the vehicle, Therein, the computing system (2) is used to define a maximum speed of the vehicle (1) during the remote operation depending on the probability of the accident and depending on the quality information.

2. The method according to claim 1, It is characterized in that The computing system (2) is used for: - determining a respective probability of each of one or more potential events occurring; and - selecting said incident from said one or more potential events depending on said respective probability of said occurrence.

3. The method according to claim 1, It is characterized in that The computing system (2) is used for: - receiving first sensor data of the sensor data from a sensor system (4) of the vehicle (1); and / or - receiving second sensor data from a vehicle-to-vehicle interface (5) of the vehicle (1).

4. The method according to any one of claims 1 to 3, It is characterized in that The computing system (2) is used for: - determining obstacle information relating to potential obstacles (1a, 1b, 6) in the environment involved in the accident based on the sensor data; and - defining said at least one parameter for said remote operation depending on said obstacle information.

5. The method according to claim 4, It is characterized in that The obstacle information includes the distance of the obstacle (1a, 1b, 6) from the vehicle (1) and / or the speed of the obstacle (1a, 1b, 6).

6. The method according to claim 4, It is characterized in that The obstacle information includes the category of the obstacle (1a, 1b, 6).

7. The method according to claim 4, It is characterized in that The computing system (2) is used to read a database in order to define the at least one parameter for the remote operation, wherein the database relates the quality information and the obstacle information to the at least one parameter.

8. The method according to any one of claims 1 to 3, It is characterized in that The predicted quality information includes a predicted end-to-end delay of the communication network (3), a predicted data transmission rate of the communication network (3) and / or a predicted packet error rate of the communication network (3).

9. The method according to any one of claims 1 to 3, It is characterized in that The computing system (2) is used for: - identifying that the accident has occurred or occurred depending on further sensor data; - initiating remote operation of the vehicle (1) via the communication network (3); as well as - controlling the vehicle (1) at least partially automatically based on the defined at least one parameter and depending on the actual quality of the communication network (3).

10. The method according to claim 9, It is characterized in that The computing system (2) is configured to control the vehicle (1) at least partially automatically based on the defined at least one parameter if the value of the actual quality of the communication network (3) is less than a predefined threshold value.

11. An electronic vehicle navigation system for a remotely operable vehicle (1), the vehicle navigation system (7) comprising a computing system (2) configured to receive sensor data relating to the environment of the vehicle (1), in, The computing system (2) is configured to: - determining the probability of an accident affecting the control of the vehicle (1) based on the sensor data; - receiving quality information (3) relating to a predicted quality of a communication network for remote operation of the vehicle, wherein the quality information corresponds to predicted quality of service information; - defining at least one parameter for the remote operation of the vehicle (1) depending on the probability of the accident and depending on the quality information, wherein defining the at least one parameter for the remote operation of the vehicle comprises defining constraints for operating the vehicle, wherein the computing system (2) is used to define a maximum speed of the vehicle (1) during the remote operation depending on the probability of the accident and depending on the quality information.

12. The electronic vehicle navigation system according to claim 11, It is characterized in that The vehicle navigation system (7) comprises: - a sensor system (4) for the vehicle (1), configured to generate first sensor data of the sensor data; and / or - a vehicle-to-vehicle interface (5) configured to receive second sensor data of the sensor data.

13. Computer program product comprising instructions which, when executed by a computer system, cause the computer system to implement the method according to one of claims 1 to 10.

Images