Collectively obtaining information for automated driving-control functions

Abstract

The invention relates to a method for monitoring an automated driving-control function, wherein data from other road users regarding their planned trajectories and positions can be transmitted to a communication interface (5) of a vehicle (3), and a computing unit (7) executes the automated driving-control function by means of a dynamic environment model continuously updated with information which is determined from data from onboard sensors and from any available data from the other road users after fusion with the data from the onboard sensors, and wherein, in order to form a dissimilarity metric, results of the dynamic environment model with respect to the positions and / or planned trajectories of the road users are compared with the ego-positions and / or planned ego-trajectories transmitted from the other road users.

Description

[0001] COLLECTIVE INFORMATION COLLECTION FOR AUTOMATED DRIVE CONTROL FUNCTIONS

[0002] The invention relates to a method for monitoring an algorithm for executing an automated driving control function of an automated vehicle, as well as a system for assisting an automated vehicle.

[0003] Automated driving control functions are known in the state of the art, not only to assist a driver in manual vehicle control for greater comfort or a higher level of safety, but in particular to replace manual vehicle control with fully automated vehicle control, ideally to enable autonomous ferry operation that does not require the assistance of a manual driver. However, the quality of such an algorithm used for the automated driving control function is largely dependent on the quality of sensor data and other information about the current situation and surroundings of the vehicle. If only the vehicle's own sensors are used, the information received is naturally limited by the fact that the vehicle's own sensors share a common perspective of the surroundings and may detect optically hidden objects, other road users, etc.cannot capture. To increase information quality and density, it is known in the art to fuse sensor data from different sources.

[0004] In this regard, US 2021 / 0063546 A1 concerns distributed sensor calibration and the sharing of sensor information via a V2X interface. In particular, it is proposed that a corresponding interface on a vehicle receive calibration data from an external source. Corresponding coordinate system transformations allow, in particular, the transfer of the position and orientation of a detected object from one vehicle to another.

[0005] It is an object of the invention to make the operation of an automatic driving control function safer.

[0006] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims.

[0007] A first aspect of the invention relates to a method for monitoring an algorithm for executing an automated driving control function of an automated vehicle, wherein data from other road users about their respective planned trajectories and respective self-determined positions can be transmitted to a communication interface of the automated vehicle, and an automated driving control function is simulated or actually executed by a computing unit of the automated vehicle with the aid of a dynamic environment model, and the environment model is continuously updated on the basis of information which is determined from data from the vehicle's own sensors and any available data from the other road users after merging with the data from the vehicle's own sensors,and wherein, to form a dissimilarity metric, a comparison of results from the dynamic environmental model regarding the respective position and / or planned trajectory of the respective road users is carried out with the own positions and / or own planned trajectories transmitted by the other road users, and if the dissimilarity metric exceeds a predetermined limit, the takeover of vehicle control by a manually controlling driver is requested and / or a history of the journey section is stored with the information for the dynamic environmental model for the later adaptation of parameters of the dynamic environmental model.

[0008] The automated driving control function serves either as a driver assistance system for intervening in manual vehicle control, or for fully automated vehicle control if an automated driving mode or even fully autonomous vehicle control is desired. While a driver assistance system supports a driver in manual vehicle control, for example, by initiating emergency braking, correcting lane guidance, or similar, an automated driving mode or autonomous vehicle control involves the takeover of driving control by the driving control function in such a way that manual vehicle control is not necessary, at least for a certain period of time or for the entire journey.

[0009] If the automated driving control function is carried out in real life, the driver assistance system will intervene in the manual control of the vehicle or there will be a period of automated vehicle control, i.e. the vehicle will be controlled mechanically. If, on the other hand, an automated driving control function is carried out in simulated form, the respective intervention by the driver assistance system or the fully automated vehicle control is only simulated. In such a simulated execution of the automated driving control function, the focus is less on checking the controller output of an open-chain setpoint follower controller, as is used, for example, in a lane departure warning system, and more on decisions and planned trajectories at higher levels, which typically specify the commands to such a setpoint follower controller.Decisions and planned trajectories for the vehicle are preferably formed by the computing unit based on environmental observation and the actual and expected behavior of other road users. This allows the decision history of the human driver and the automated driving control function to be compared during manual operation of the vehicle.

[0010] This results in a multi-stage process by the computing unit to both obtain data for the dynamic environment model and to determine the dissimilarity metric. For example, if the automated vehicle is traveling in a barely developed hinterland without encountering other road users, it relies on the data from its own sensors (assuming no other data sources are available, such as unmanned aircraft or data from satellites monitoring the automated vehicle's current location).

[0011] However, if other road users are in the vicinity of the automated vehicle, it is possible to exploit the fact that various and sometimes disparate sensor sources, when observing a local situation, are generally capable of obtaining more reliable and comprehensive information about the traffic situation and the surroundings than the sensors of a single vehicle. This circumstance is exploited in that the vehicles are networked with each other, i.e., they have a so-called V2X interface to exchange information with each other. The relevant information for implementing the first aspect of the invention is, in particular, the respective position of a vehicle, as determined by the vehicle's own sensors, and a planned trajectory of the respective vehicle (i.e., road users).

[0012] While, in the presence of other vehicles, the automated vehicle under consideration not only continuously estimates the positions of the other road users with the help of the dynamic environment model, but also projects their current behavior into the future and algorithmically estimates the behavior of the respective other road users in the future, this information can also be transmitted from the other road users themselves to the automated vehicle under consideration via the above-mentioned V2X interface.

[0013] While each of the vehicles manages and updates its own dynamic environment model, in the presence of other road users with appropriate capabilities and a V2X interface, it can support its own updated dynamic environment model and determine corresponding confidences of the data from the vehicle's own sensors.

[0014] In this context, a central location can be used, which, instead of the peer-to-peer connection between the individual road users, acts as a server to merge all positions and planned trajectories of the individual road users into a collective overall picture. This collective overall picture can, in turn, be transmitted to each of the vehicles, enabling each of the vehicles to create a respective dissimilarity metric and compare it with corresponding threshold values.

[0015] If there are significant discrepancies between the driver's own perception of the environment and the behavior of road users within the collective overall picture, this is an indication that an algorithm for creating the dynamic environment model needs to be adjusted, or even that a sensor in the driver's own vehicle may be defective. If the resulting dissimilarity metric indicates this, the entirety of all available information is collected as an approximation of the so-called "ground truth" in order to re-parameterize the respective dynamic environment model. In the case of an artificial neural network used to capture the environment, perform image recognition, or make decisions regarding the current traffic situation based on estimated and recorded behavior of other road users or objects in the current traffic situation (such as traffic signs, warning beacons, traffic lights, construction sites, etc.), the algorithm is used to calculate the behavior of other road users or objects in the current traffic situation.), in particular the weights of the artificial neural network are adjusted.

[0016] Thus, sensor fusion from different sources allows for easier validation and adaptation of algorithms used to execute an automated driving control function, since, as explained above, the ideal of "ground truth" is approximated with the increasing number of sources of information and thus the so-called "supervised learning" of adaptive systems can be simplified, in particular through rigorous analysis of the algorithms used and the cause of the occurrence of an error.

[0017] As an alternative to the validation, verification, adaptation and / or testing of the algorithms used to execute the automatic driving control function as described above, the takeover of an automatically operated vehicle by a human driver can also be encouraged in real time in order to exploit the strengths of a human driver in particularly difficult situations and to resolve the driving situation.

[0018] Data from other road users regarding their planned trajectories and self-determined positions are transmitted to the communication interface of the automated vehicle, especially if appropriately equipped vehicles are present in the immediate vicinity of the automated vehicle in question. In this case, the other road users, in particular other vehicles, transmit the trajectories determined according to their own trajectory planning, as well as their own position determined for themselves. This data is accordingly data on the position and the planned trajectory, which are each determined from the first-person perspective of the other road users.

[0019] While the respective determined positions are based purely on measurements, the trajectories are planned paths and, in a broader sense, part of the behavior of other road users. However, these same data are also determined and estimated from the ego perspective of the automated vehicle under consideration with respect to other road users, and are thus comparable with the data of the other road users. Against this background, the dissimilarity metric is determined in such a way that it takes this difference into account and determines the quality of the assessment of the position and planned trajectories of the other road users by the vehicle's own sensors by comparing them with the corresponding data of the other road users themselves.

[0020] If not only data from other road users is available, but also environmental data from stationary sensors or other sources such as drones or satellite images, the determination of the dissimilarity metric is preferably carried out based on these additional data sources, which provide further data to approximate the "ground truth" and thus make the decisions of the algorithms for executing the automated driving control function more verifiable. In this sense, the additional data from other sources such as stationary sensors, drones, or satellite images can also be used to adapt the algorithms for executing the automated driving control function if the values ​​of the dissimilarity metric provide reason for doing so.

[0021] An advantageous effect of the invention is that not only the data from the vehicle's own sensors are used to determine inconsistencies in the estimation of the traffic situation by the dynamically updated environmental model, but also, incrementally, as much data from other sources as possible is used. These other sources are, in particular, other road users such as other vehicles, but, if available, also stationary sensors, unmanned aircraft, satellite information, etc. While the vehicle's own sensors can only ever capture the environment from one and the same vehicle-specific perspective, these external sources provide data about the same situation, but from a different perspective. As the number and quality of the external sources increase, an overall environmental image can be determined that is closest to the so-called "ground truth," i.e.increasingly approximates the objective and actual characteristics of the traffic situation and the environment around the vehicle under consideration. This makes it easier to identify inadequate behavior of the driving control function, which is based in particular on an incorrect assessment by the automated vehicle regarding the environment and the traffic situation. If the automated vehicle itself, in particular its processing unit, incorrectly assesses a position or expected behavior of a road user, the error can be identified by transmitting the position and planned trajectory of the road user, while the data determined from a road user regarding their environment and position are in turn transmitted to the automated vehicle under consideration.

[0022] According to an advantageous embodiment, the history is used after completion of the journey of the automated vehicle for analyzing and / or adapting the algorithm for executing the automated driving control function in order to adapt the parameters of the dynamic environment model using the history for the input data of the dynamic environment model with predetermined output data.

[0023] According to an advantageous embodiment, a method is therefore provided for adapting an algorithm for executing an automated driving control function of an automated vehicle, wherein data from other road users about their respective planned trajectories and respective self-determined positions can be transmitted to a communication interface of the automated vehicle, and an automated driving control function is simulated or actually executed by a computing unit of the automated vehicle with the aid of a dynamic environment model, and the environment model is continuously updated on the basis of information which is determined from data from the vehicle's own sensors and any available data from the other road users after merging with the data from the vehicle's own sensors,and wherein, to form a dissimilarity metric, a comparison of results from the dynamic environmental model regarding the position and / or planned trajectories of the respective road users is carried out with the positions and / or planned trajectories transmitted by the other road users, and if the dissimilarity metric exceeds a predetermined limit, a history of the journey section is stored with the information for the dynamic environmental model and, after completion of the journey of the automated vehicle, is used to analyze and / or adapt the algorithm for executing the automated driving control function.

[0024] According to a further advantageous embodiment, the data possibly transmitted by other road users to the communication interface of the system comprise information about the surroundings of the other road users as detected by the respective other road users.

[0025] According to this embodiment, the other road users not only determine their own position and their planned trajectories, but also information about their surroundings and other road users, including the automated vehicle under observation, from their own perspective. The algorithms for executing the automated driving control function can thus check whether their own planned trajectory and their own determined position match the perspective of the other road users. If there is no match, a dangerous situation can arise, as this indicates a misjudgment of the automated vehicle under observation by the other road users.

[0026] According to a further advantageous embodiment, the data transmitted by other road users and received at the communication interface are merged with the data from the vehicle's own sensors and statistically averaged.

[0027] According to a further advantageous embodiment, the updating of the dynamic environment model is additionally carried out on the basis of information which is determined by a stationary environment detection unit and which is fused with the data from the vehicle's own sensors, and wherein a comparison of results of the dynamic environment model with the data of the environment detection unit is carried out to form the dissimilarity metric.

[0028] The stationary environmental detection unit is formed, for example, by one or more cameras on masts that monitor the traffic situation at least in a specific area. Alternatives to the cameras are, of course, possible, such as radar devices.

[0029] According to a further advantageous embodiment, a stationary control center is used to collect respective data from on-board sensors and planned trajectories of a plurality of vehicles, wherein the control center stores additional environmental information and / or traffic information of the situation and environment during which the dissimilarity metric has exceeded the predetermined limit value and provides it as input data and / or output data of the dynamic environmental model.

[0030] By using such a central location, data from the respective road users and vehicles can be collected centrally and distributed easily using technical means. This simplifies the creation of a collective overall picture of the current traffic situation and the exchange of information about the planned trajectories and positions of individual road users, especially vehicles. This makes it particularly easy to check for inconsistencies in the situational awareness of individual units.

[0031] According to a further advantageous embodiment, the algorithm for executing an automated driving control function comprises an artificial neural network, wherein the parameters comprise weights of the artificial neural network.

[0032] An artificial neural network is typically trained with sets of predefined input data and predefined output data. This can be used, for example, to perform machine object recognition in camera images. However, if the input data deviates from the predefined input data in reality, the artificial neural network can determine completely different output data, although this may not seem justified to the human observer. In such a case, the artificial neural network must be adapted accordingly, and an additional learning process performed. Any data obtained outside the ego perspective of the automated vehicle being observed can be used for this purpose, in particular from other road users in the vicinity of the automated vehicle, and possibly also from a stationary environmental detection unit such as stationary cameras.

[0033] According to a further advantageous embodiment, confidences of the data from the vehicle's own sensors are determined and compared with further confidences determined from the data of other road users that are received and merged with the data from the vehicle's own sensors.

[0034] A further aspect of the invention relates to a system for assisting an automated vehicle, comprising a communication interface for receiving data from other road users, and a computing unit designed to simulate or actually execute an automated driving control function of the automated vehicle while driving, as well as to use a dynamic environment model to execute the automated driving control function and to continuously update it on the basis of information that is determined by the computing unit from the data of the vehicle's own sensors and, if available, from data sent by other road users and received via the communication interface about planned trajectories and positions of the other road users, each as planned and determined by the other road users, after merging with the data from the vehicle's own sensors.and wherein the computing unit is designed to perform a comparison of results of the dynamic environmental model regarding the position and / or planned trajectories of the road users based on the data from the vehicle's own sensors with the planned trajectories and / or positions transmitted by the other road users in order to form the dissimilarity metric, and, in the event of the dissimilarity metric exceeding a predetermined limit, to request that a manually controlling driver take over control of the vehicle and / or to store a history of the journey section with the information for the dynamic environmental model for later adaptation of parameters of the dynamic environmental model.

[0035] According to a further advantageous embodiment, the computing unit is designed to merge the data transmitted by other road users and received at the communication interface with the data from the vehicle's own sensors and to statistically average them. According to a further advantageous embodiment, the computing unit is designed to update the dynamic environment model based on information determined by a stationary environment detection unit and merged with the data from the vehicle's own sensors, and wherein the computing unit is designed toTo form the dissimilarity metric, a comparison of the results of the dynamic environment model with regard to the respective position and / or planned trajectory of the respective road users with the own positions and / or own planned trajectories transmitted by the other road users, merged with the data of the environment detection unit, and in the event of the dissimilarity metric exceeding a predetermined limit, the takeover of vehicle control by a manually controlling driver is requested and / or a history of the journey section with the information for the dynamic environment model is stored for the later adaptation of parameters of the dynamic environment model.

[0036] According to a further advantageous embodiment, the computing unit has an adaptation interface to which the history with the information for the dynamic environment model can be fed in in order to adapt parameters of the dynamic environment model with predetermined output data using the history for the input data of the dynamic environment model.

[0037] According to a further advantageous embodiment, the system further comprises a stationary control center for collecting respective data from on-board sensors and planned trajectories of a plurality of vehicles, wherein the control center is designed to store additional environmental information and / or traffic information of the situation and environment during which the dissimilarity metric has exceeded the predetermined limit value and to provide it as input data and / or output data of the dynamic environmental model.

[0038] According to a further advantageous embodiment, the dynamic environment model comprises an artificial neural network, wherein the parameters comprise weights of the artificial neural network.

[0039] According to a further advantageous embodiment, the computing unit is designed to determine confidences of the data from the vehicle's own sensors and to compare them with further determined confidences from the data of other road users that have been received and merged with the data from the vehicle's own sensors.

[0040] A further aspect of the invention relates to a vehicle with a system as described above and below.

[0041] Advantages and preferred developments of the proposed system result from an analogous and analogous transfer of the statements made above in connection with the proposed method.

[0042] Further advantages, features, and details will become apparent from the following description, which – where appropriate with reference to the drawings – describes at least one embodiment in detail. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0043] They show:

[0044] Fig. 1 : A traffic situation in which a method according to an embodiment of the invention for monitoring an algorithm for executing an automated driving control function of an automated vehicle is applied.

[0045] Fig. 2: A vehicle implementing the method described in Fig. 1.

[0046] The representations in the figures are schematic and not to scale.

[0047] Fig. 1 shows a traffic situation at an intersection in which the automated vehicle 3 under consideration is driving to the left in a turning lane, while other road users have also entered the intersection area or are just leaving it again. For the execution of the algorithm for executing the automated driving control function, it is crucial to record the positions of the other road users, their planned trajectories, and traffic signs. For this purpose, a method for monitoring the algorithm is executed in the vehicle 3. Such a vehicle 3 with its corresponding components is shown in Fig. 2, which is why the following description refers to both Fig. 1 and Fig. 2 without using a separate description of Fig. 2.Essentially, the automated vehicle 3 under consideration has a multitude of its own sensors, which, however, have in common that they can only observe their environment from the ego perspective of the automated vehicle 3. Not only for the correct execution of the algorithm for executing the driving control function, but also for strict monitoring of the algorithm, as much external data as is available in the given situation is gradually used. In most cases, this is achieved through the increasing establishment of V2X communication using data from other road users; however, data from stationary sensor units can also be used if available. For this purpose, data from other road users about their respective planned trajectories and their respective self-determined positions are transmitted to a communication interface 5 of the automated vehicle 3.A computing unit 7 of the automated vehicle 3 further executes an automated driving control function with the aid of a dynamic environmental model. The dynamic environmental model models, in particular, the position and behavior of other road users, especially their trajectories. The environmental model is continuously updated based on information obtained from data from the vehicle's own sensors and the available data from other road users after merging with the data from the vehicle's own sensors. To generate a dissimilarity metric, the results of the dynamic environmental model regarding the respective position and planned trajectory of the respective road users are then compared with the vehicle's own positions and planned trajectories transmitted by the other road users.If the dissimilarity metric exceeds a predefined threshold, a history of the journey segment is stored with the information for the dynamic environment model for later adjustment of the dynamic environment model's parameters. After the automated vehicle 3 has completed its journey, the history is used to analyze and subsequently adapt the algorithm for executing the automated driving control function, adjusting the algorithm's parameters to better handle such situations in the future.

[0048] Although the invention has been illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned by way of example are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

[0049] List of reference symbols

[0050] 1 system

[0051] 3 vehicles

[0052] 5 Communication interface

[0053] 7 Computing unit

Claims

Patent claims 1. Method for monitoring an algorithm for executing an automated driving control function of an automated vehicle (3), wherein data from other road users about their respective planned trajectories and self-determined positions can be transmitted to a communication interface (5) of the automated vehicle (3), and an automated driving control function is simulated or actually executed by a computing unit (7) of the automated vehicle (3) using a dynamic environment model, and the environment model is continuously updated on the basis of information obtained from data from the vehicle's own sensors and the data of other road users that may be available after fusion with the data from the vehicle's own sensors,and wherein, to form a dissimilarity metric, a comparison is made between the results of the dynamic environment model regarding the respective position and / or planned trajectory of the respective road users and the positions and / or planned trajectories transmitted by the other road users, and in the event that the dissimilarity metric is exceeded by a predefined limit, the takeover of vehicle control by a manually controlling driver is requested and / or a history of the journey segment with the information for the dynamic environment model is saved for the later adjustment of parameters of the dynamic environment model.

2. Method according to claim 1, wherein the history after the completion of the journey of the automated vehicle (3) is used to analyze and / or adapt the algorithm for executing the automated driving control function in order to adapt the parameters of the dynamic environment model using the history for the input data of the dynamic environment model with predetermined output data.

3. Method according to one of the preceding claims, wherein the data optionally transmitted by other road users to the communication interface (5) of the system (1) includes information about the environment of the other road users as perceived by the respective other road users. A method according to any one of the preceding claims, wherein the data transmitted by other road users and received at the communication interface (5) are fused with the data from the vehicle's own sensors and statistically averaged. A method according to any one of the preceding claims, wherein the update of the dynamic environment model is additionally performed on the basis of information determined by a stationary environment detection unit and fused with the data from the vehicle's own sensors, and wherein a comparison of results of the dynamic environment model with the data of the environment detection unit is performed to form the dissimilarity metric.A method according to claim 5, wherein a stationary central unit is used to collect data from vehicle-specific sensors and planned trajectories of a plurality of vehicles, the central unit storing and storing additional environmental and / or traffic information about the situation and environment during which the dissimilarity metric exceeded the predetermined limit value and providing it as input and / or output data for the dynamic environment model. A method according to any of the preceding claims, wherein the algorithm for executing an automated driving control function comprises an artificial neural network, the parameters of which comprise weights of the artificial neural network.A method according to one of the preceding claims, wherein confidences of the data from the vehicle's own sensors are determined and compared with further determined confidences from the data of other road users obtained and fused with the data from the vehicle's own sensors. A system (1) for supporting an automated vehicle (3), comprising a communication interface (5) for receiving data from other road users, and a computing unit (7) configured to simulate or actually execute an automated driving control function of the automated vehicle (3) during driving, and to use a dynamic environment model for executing the automated driving control function based on. to continuously update information obtained by the computing unit (7) from the data of the vehicle's own sensors and, if available, from data transmitted by other road users and received via the communication interface (5) concerning planned trajectories and positions of the other road users, as planned and determined by the other road users, after fusion with the data from the vehicle's own sensors, and wherein the computing unit (7) is configured to perform a comparison of results of the dynamic environment model for the position and / or planned trajectories of the road users based on the data of the vehicle's own sensors with the planned trajectories and / or positions of itself transmitted by the other road users, in order to form the dissimilarity metric.and, in the event that the dissimilarity metric exceeds a predetermined limit, to request the takeover of vehicle control by a manually controlling driver and / or to save a history of the journey segment with the information for the dynamic environment model for the subsequent adjustment of parameters of the dynamic environment model. Vehicle (3) with a system (1) according to claim 9.

Images