Computer-implemented method and device for generating training data

Abstract

A computer-implemented method for generating training data, the method comprising the following steps: - Providing (S1) driving data of an ego vehicle; - Calculating (S2), in particular for a given current time, a set of possible behaviors for the ego vehicle based on the driving data using a behavior planner algorithm; - Select (S3) at least one of the possible behaviors based on a selection criterion; - Generating (S4) at least one linguistic representation of the selected at least one behavior based on information contained in the selected at least one behavior; - Adding (S5) at least one linguistic representation to driving data to generate the training data; and - Providing (S6) the generated training data for training a machine learning model.

Description

[0001] The invention relates to a computer-implemented method for generating training data. The invention further relates to a device for generating training data and a method for training a machine learning model to provide at least one autonomous driving function or a driver assistance function of a vehicle. State of the art

[0002] The development of automated driving functions is experiencing significant progress through the use of machine learning (ML). Recently, approaches have emerged that utilize foundation models (FM) for behavior planning. The integration of language as an additional input source, as seen in large language models (LLM), is proving particularly promising. This method can enhance functional performance and also offers ways to explain the output of machine learning models, as demonstrated by Wayve's LINGO-1.

[0003] A key challenge with this approach is the availability of voice data needed to train the models. This data must describe in detail what happens in each driving scenario, such as "the ego vehicle brakes because a pedestrian is crossing the road in front of it." To collect this data, Wayve, for example, encouraged its drivers to comment on their driving actions and record these comments.

[0004] It is an object of the invention to provide an improved method and / or an improved device.

[0005] The problem is solved by a method according to the features of claim 1. The problem is solved by a device according to the features of claim 10. Disclosure of the invention

[0006] According to a first aspect, a computer-implemented method for generating training data is proposed. The method comprises the following steps: - Providing driving data from an ego vehicle; - Calculating, especially for a given current point in time, a set of possible behaviors for the ego-vehicle based on the driving data using a behavior planner algorithm; - Selecting at least one of the possible behaviors based on a selection criterion; - Generating at least one linguistic representation of the selected at least one behavior based on information contained in the selected at least one behavior, in particular by the behavior planner algorithm; - Adding at least one linguistic representation to driving data to generate the training data; and - Providing the generated training data to train a machine learning model.

[0007] It is understood that the steps according to the invention, as well as further optional steps, do not necessarily have to be carried out in the sequence shown, but can also be carried out in a different sequence. Furthermore, additional intermediate steps may be provided. The individual steps may also comprise one or more sub-steps without thereby departing from the scope of the method according to the invention.

[0008] According to a second aspect, a device for generating training data is proposed. The device includes an evaluation and computing unit designed to perform the following steps: - Providing driving data from an ego vehicle; - Calculating, especially for a given current point in time, a set of possible behaviors for the ego-vehicle based on the driving data using a behavior planner algorithm; - Selecting at least one of the possible behaviors based on a selection criterion; - Generating at least one linguistic representation of the selected at least one behavior based on information contained in the selected at least one behavior, in particular by the behavior planner algorithm; - Adding at least one linguistic representation to driving data to generate the training data; and - Providing the generated training data to train a machine learning model.

[0009] Optionally, the procedure includes filtering of unwanted / non-compliant data in relation to the behavior planner algorithm.

[0010] The statements made regarding the procedure apply accordingly to the device. It is understood that linguistic modifications of procedurally formulated features can be reformulated for the device according to common linguistic practice, without such formulations needing to be explicitly listed here.

[0011] The present invention proposes a method that automates the generation of the required speech data, thereby eliminating the need for manual data creation, either entirely or partially. Furthermore, the data must be curated for training a machine learning model. For a behavior planner in particular, it is crucial that the training data contains only rule-compliant expert trajectories, i.e., trajectories of drivers who adhere to all traffic regulations. The present invention therefore also provides a method for checking the data to ensure that only desired, rule-compliant data is used for training, while undesirable data is filtered out.

[0012] The present invention uses a behavior planner algorithm, particularly a geometric one, for generating speech data and preferably for checking driving data or input data (especially trajectories) of the ego-vehicle. The behavior planner algorithm preferably calculates a set of possible behaviors for the ego-vehicle for the current time.

[0013] These preferably comprise geometric areas (condition, go, and no-go zones). Condition and go zones preferably include constraints (e.g., the time period during which a specific zone may be entered), while no-go zones may not be entered at all. Additionally, the reasons why the behavior planner algorithm calculated the zones are preferably defined (e.g., that a time-limited condition zone was created in the opposite lane to allow a blockage in one's own lane to be avoided). This information This information can be reconstructed or provided, in particular, from a node structure of the behavior planner algorithm, which may include a behavior mixing tree. According to the invention, this information is automatically converted into a linguistic representation that describes the driving scene and preferably justifies actions. This can be done directly during the drive or by re-simulating existing measurements without speech data. Manual generation of the speech data is therefore unnecessary.

[0014] Alternatively, the system can be used for pre-labeling, meaning it automatically suggests language labels, and a human then preferably reviews and adjusts the labels as needed. This results in time savings compared to fully manual labeling.

[0015] In particular, when processing existing measurements, the information calculated by the behavior planner algorithm is preferably used, according to the invention, to check the expert trajectories (or at least various sections of the trajectories) present in the measurements for their rule compliance. An example of non-compliant data is when the trajectory exceeds the permitted maximum speed. If a trajectory, for example, meets the requirements of a high-priority behavior (Geometric Behaviors), it has a high rule compliance and is selected as suitable. Data with selected trajectories can then preferably be used, for example, to train a behavior planner. The data check preferably does not take place during driving, but offline on existing measurements.The generation of the linguistic representation is performed as additional, linguistically represented data for training a machine learning model. The generation of a linguistic representation and the validation and selection of data can be done individually (e.g.,

[0016] The system can be used for generating language labels for driving data, selecting compliant driving data, as well as in combination (e.g., generating data sets from compliant driving data with language labels).

[0017] The text label or linguistic representation is preferably added to the original driving data and can be used, for example, for training a machine. Learning models can be used. As an alternative to using the text labels directly, the labels can be reviewed and, if necessary, corrected by a human before further use. If the behavior planner algorithm is used to check data in the case of recorded measurements, this check takes place in

[0018] The next step is "behavior selection." This involves checking, preferably for the current ego trajectory, whether it meets the requirements of one or more of the behavioral alternatives. If so, the trajectory is considered rule-compliant. If the trajectory is considered valid, it is not considered to be in accordance with the rules and can be removed from the data set, for example.

[0019] As an alternative to selecting a single behavioral option, several or all behavioral options can be shared and converted into a linguistic representation (optionally including the estimated probability that each behavioral option can be applied). Sharing the text labels for multiple behavioral options can be particularly advantageous when the labels are presented to a human for review. The human can then select the most appropriate text label.

[0020] One advantage of the method is its close coupling to the behavior planner algorithm approach, ensuring that the actual decision and the explanation used for training are consistent.

[0021] Another aspect proposes that the driving data be captured in real time by the ego-vehicle, and / or provided by driving measurements from a past trip with the ego-vehicle, and / or provided by simulating a past trip with the ego-vehicle; and / or generated synthetically through data augmentation. Furthermore, other methods for creating synthetic data can also be used. For example, data generated from a simulation can also be used.

[0022] The system uses driving data or scenario data as input. This can include data generated live in the vehicle or captured by sensors, as well as previously recorded measurements that are replayed or re-simulated. The system preferably generates appropriate language labels or linguistic representations for this data. Alternatively, synthetically generated driving data (e.g., from relevant simulation environments) can also serve as input.

[0023] Another aspect is proposed: that at least one linguistic representation should contain a description of a driving situation and / or a justification for a behavioral alternative.

[0024] The data is processed by the behavior planner algorithm, which calculates the behavioral alternatives (geometric behaviors) for the current time. A more suitable behavior is then selected and passed on along with the corresponding information. The information contained in the selected behavior is then converted into a linguistic representation. This representation may include a description of the situation and justifications for the behavioral alternatives, and preferably follows a defined formalism for scene description.

[0025] In another aspect, it is proposed that the selection criterion includes a check by the behavior planner algorithm to determine whether a, in particular current, ego trajectory of the ego vehicle meets requirements for one or more of the behavioral alternatives.

[0026] If critical scenes in the dataset are to be identified where the driver did not behave in accordance with the rules (e.g., due to an emergency maneuver), the trajectory check can be adjusted to select trajectories that do not meet the requirements of any of the behavioral alternatives, or only those with a priority below a given threshold. If it is also necessary to consider whether the maneuver was justified as an emergency or simply a driving error, it can be checked whether valid trajectories existed for higher-priority behaviors. For this purpose, trajectories can be generated for the scene using a classic planner (e.g., the tactical planner of the behavior planner algorithm). If these trajectories also do not meet any higher-priority behavioral alternatives than the expert trajectory, this indicates that the expert trajectory is not rule-compliant for good reason (e.g., because it is not a valid behavioral alternative).(due to an emergency maneuver) and the scene is selected as a critical scene.

[0027] However, if the trajectories of the classical planner fulfill higher-priority behavioral alternatives than the expert trajectory, the future development of the scene can still be examined to, for example, rule out errors resulting from suboptimal perception. If a reason for the violation of rule conformity is found (e.g., a critical object that perception recognized too late), the expert trajectory is preferably selected as valid, and the scene is retained as a critical scene. If no plausible reason is found, the scene is preferably discarded, since the expert behaved non-conformingly for no reason, and the scene can therefore be considered neither rule-compliant nor an interesting critical scene.

[0028] Annotating the data with voice labels can preferably be done online in the vehicle, for example, by having the behavior planner algorithm provide voice output commentary on the events, and allowing a human to mark and / or correct potential errors and / or inaccuracies, e.g., by pressing a button or giving a voice command. The behaviors identified as non-compliant can preferably be collected in a cloud or on a server. From the resulting knowledge base or database, it is possible to extract implicit behaviors within selected ODDs (e.g., of a country).

[0029] In another aspect, it is proposed that the trajectory test be extended to include a condition that the trajectory meets requirements for behavioral alternatives with the highest priority or meets behavioral alternatives with a priority greater than a threshold.

[0030] To adjust and ensure the quality of the generated dataset, the trajectory check can be extended to include the condition that the trajectory meets the requirements of the N highest-priority behavioral alternatives (or alternatively, only behavioral alternatives with a priority greater than a threshold). If only the requirements of lower-priority behavioral alternatives are met, the trajectory can, for example, be rejected.

[0031] In another aspect, it is proposed that at least one linguistic representation includes natural language text and / or a non-natural language encoding of a text into defined character strings.

[0032] The generated linguistic representation can be normal, natural language text as described above. Alternatively, the text can also be a corresponding, non-natural language encoding of the text into defined character strings. As an alternative or supplement to the purely linguistic representation, the generated label can contain additional information, such as an image of the scene with the condition and / or go and / or no-go areas marked. More generally, it can also involve additional input features. Vision-language models, for example, can process language and images and then generate a future trajectory based on this information.

[0033] In another aspect, a method for training a machine learning model to provide at least one autonomous driving function or a driver assistance function of a vehicle is proposed, wherein the machine learning model is trained using the training data generated according to the method in accordance with one of its aspects.

[0034] The invention can be used in machine learning-based driver assistance systems and in autonomous driving.

[0035] In another aspect, a method for providing at least one autonomous driving function or one driver assistance function of a vehicle is proposed using a trained machine learning model, wherein the machine learning model was trained using the training data generated according to one of the aspects of the method.

[0036] The driving data (trajectories and preferably all other available data, e.g. sensor data) and the generated language labels can preferably be used to train, for example, a machine learning model (e.g., a Foundation Model), or to fine-tune a pre-trained Foundation Model for a specific task.

[0037] In another aspect, a control unit is also claimed, which is included in a vehicle with an autonomous driving function and / or a robotics system, and on which the present method for generating the training data or the method for training the machine learning model or a method for providing an autonomous driving function or a driver assistance function by means of a machine learning model trained on the basis of the generated training data can be executed in one of its aspects.

[0038] In another aspect, a computer program is claimed to contain program code capable of executing at least parts of the present method in one of its aspects when the computer program is executed on a computer. In other words, a computer program (product) is claimed to comprise instructions that, when executed by a computer, cause it to execute the method(s) in one of its aspects.

[0039] In a further aspect, a computer-readable data carrier containing the program code of a computer program is proposed to execute at least parts of the present method in one of its aspects when the computer program is executed on a computer. In other words, the invention relates to a computer-readable (storage) medium comprising instructions which, when executed by a computer, cause it to execute the method / steps of the method in one of its aspects.

[0040] The described configurations and training programs can be combined in any way desired.

[0041] Further possible embodiments, developments and implementations of the invention also include combinations of features of the invention described previously or subsequently with regard to the exemplary embodiments that are not explicitly mentioned. Brief description of the drawings

[0042] The accompanying drawings are intended to provide a further understanding of the embodiments of the invention. They illustrate embodiments and, in conjunction with the description, serve to explain the principles and concepts of the invention.

[0043] Other embodiments and many of the aforementioned advantages become apparent with reference to the drawings. The elements depicted in the drawings are not necessarily shown to scale. Fig. Figure 1 shows a flowchart of an embodiment of the present method. Fig. Figure 2 shows a block diagram of an embodiment of the present device. Fig. Figure 3 shows an example driving scene of a selected behavioral alternative.

[0044] In the figures of the drawings, identical reference symbols denote identical or functionally equivalent elements, parts or components, unless otherwise stated.

[0045] Fig. Figure 1 shows a schematic flowchart of a procedure for generating training data.

[0046] The method can be carried out in any embodiment, at least partially, by a device 100, which may comprise several components not shown in detail, for example, one or more provisioning units and / or at least one evaluation and computing unit. It is understood that the provisioning unit may be designed together with the evaluation and computing unit, or it may be different from it. Furthermore, the device 100, which may be part of a system, may comprise a storage unit and / or an output unit and / or a display unit and / or an input unit.

[0047] The computer-implemented method comprises at least the following steps when preferably performed on the device 100.

[0048] The procedure is also referred to as Fig. 2 explained.

[0049] In step S1, driving data 200 of an ego vehicle is provided.

[0050] In step S2, a calculation is performed, in particular for a current point in time, of a set of possible behaviors 202 for the ego vehicle based on the driving data 200 by a behavior planner algorithm 204.

[0051] In step S3, at least one of the possible behaviors 206 is selected based on a selection criterion.

[0052] In step S4, at least one linguistic representation 208 of the selected at least one behavior 206 is generated based on information contained in the selected at least one behavior 206. The linguistic representations 208 are preferably generated by the behavior planner algorithm 204.

[0053] In step S5, at least one linguistic representation 208 is added to driving data 200 to generate the training data 210.

[0054] In step S6, the generated training data 210 is provided for training a machine learning model 212.

[0055] Fig. Figure 3 shows an example of a behavioral alternative in which an ego vehicle 300 must navigate around a blockage caused by a parked vehicle 302. The ego vehicle 300 can navigate around the blockage within a specific timeframe. For this purpose, time constraints are set for condition areas 304 and go areas 306. The ego vehicle 300 is driving in a lane and, due to the blockage, must move into an oncoming lane 310. A no-go area 312 may exist around the parked vehicle, which the ego vehicle must not enter during the evasive maneuver.

[0056] From the behavioral alternatives, a language label of the following type is generated, for example: "The ego vehicle is approaching a blockage. The blockage is formed by a vehicle parked in the ego lane. Since there is no oncoming traffic, the ego vehicle can drive around the blockage on the left in the opposite lane. Due to the limited visibility of the opposite lane, the maneuver around the blockage must be completed within 5 seconds."

Claims

[1] Computer-implemented method for generating training data, the method comprising the steps: - Providing (S1) driving data (200) of an ego vehicle (300); - Calculate (S2), in particular for a current time, a set of possible behaviors (202) for the ego vehicle (300) based on the driving data (200) by a behavior planner algorithm (204); - Selecting (S3) at least one of the possible behaviors (206) based on a selection criterion; - Generating (S4) at least one linguistic representation of the selected at least one behavior (206) based on information contained in the selected at least one behavior (206); - Adding (S5) at least one linguistic representation (208) to driving data (200) to generate the training data (210); and - Providing (S6) the generated training data (210) for training a machine learning model (212). [2] Method according to claim 1, wherein the driving data (200) are acquired in real time by the ego vehicle (300), and / or are provided by driving measurements of a past trip with the ego vehicle (300), and / or are provided by simulation of a past trip with the ego vehicle (300); and / or are generated synthetically by data augmentation. [3] Method according to claim 1 or 2, wherein the at least one linguistic representation (208) comprises a description of a driving situation and / or a justification for a behavior alternative (202, 206). [4] Method according to one of the preceding claims, wherein the selection criterion comprises a check by the behavior planner algorithm (204) to see whether, in particular a current, ego trajectory of the ego vehicle meets requirements for one or more of the behavior alternatives (202). [5] Method according to claim 4, wherein the trajectory test is extended by a condition that the trajectory satisfies requirements for behavior alternatives (202) with the highest priority or behavior alternatives (202) with a priority greater than a threshold. [6] Method according to any of the preceding claims, wherein the at least one linguistic representation (208) comprises natural language text and / or a non-natural language encoding of a text into defined character strings. [7] Method for training a machine learning model (212) to provide at least one autonomous driving function or a driver assistance function of an ego vehicle (300), wherein the machine learning model (212) is trained using the training data generated according to the method according to any one of claims 1 to 6. [8] Computer program with program code to execute at least parts of a method according to any one of claims 1 to 7 when the computer program is executed on a computer. [9] Computer-readable data carrier containing program code of a computer program for executing at least parts of a method according to any one of claims 1 to 7 when the computer program is executed on a computer. [10] Device (100) for generating training data, wherein the device (100) has an evaluation and computing unit configured to perform the following steps: - Providing driving data (200) of an ego vehicle (300); - Calculating, especially for a given current time, a set of possible behaviors (202) for the ego vehicle (300) based on the driving data (200) by a behavior planner algorithm (204); - Selecting at least one of the possible behaviors (206) based on a selection criterion; - Generating at least one linguistic representation (208) of the selected at least one behavior (206) based on information contained in the selected at least one behavior (206); - Adding at least one linguistic representation (208) to driving data (200) to generate the training data (210); and - Providing the generated training data (210) for training a machine learning model (212).

Images