Method for generating interaction annotations for a training data set, computer-implemented methods for training a machine learning model on the perception of interaction, trajectory, and human behavior, computer-implemented methods for the perception of interaction, trajectory, and human behavior, computer program, storage medium, data carrier signal, and automated vehicle

Abstract

Method for generating interaction annotations for a training data set, wherein the training data set comprises sequences and a machine learning model can be trained using the generated annotated training data set, the method comprising the steps: • Obtaining a sequence of first data, wherein the first data are acquired by at least one first sensor (3), wherein the first data comprise information about the vehicle surroundings and / or vehicle interior and / or driving behavior (11); • Obtaining second data, wherein the second data is detected by a second sensor (2), wherein the second data comprises at least one interaction of at least one person with the automated vehicle (1), wherein the second sensor (2) detects an opening of a vehicle door, wherein the first data is synchronized with the second data (12); • Generating interaction annotations based on the at least one interaction and annotating the first data with the interaction annotations (I3); • Generating person behavior annotations, wherein the person behavior annotations characterize (P1) a person behavior before and / or after an interaction of a person with an automated vehicle (1) based on the generated interaction annotations; • Annotate the first data with the person behavior annotations (P2).

Description

The invention relates to a method for generating interaction annotations. The invention also relates to computer-implemented methods for training a machine learning model for perception of interaction, trajectory and person behavior, and to computer-implemented methods for perception of interaction, trajectory and person behavior. The invention further relates to a computer program, a storage medium, a data carrier signal and an automated vehicle.The following definitions apply to the entire disclosure content.For automated driving, in particular automated driving of automation levels 3, 4, and 5 according to SAE J3016, machine learning models are increasingly involved. In particular for automated vehicles provided for transporting persons, such as robotized systems, shuttles or other vehicles, interactions of persons with the automated vehicle provide important information. Interactions can be used to draw conclusions about preceding and subsequent scenarios.To train a machine learning model to the perception of interactions, a training dataset is required. In order to generate such a training data set with as little effort as possible, it has surprisingly been found within the scope of the invention that a reference sensor, in particular a sensor which detects an activity of a vehicle door, can advantageously be used as a trigger for interactions.The prior art is disclosed in DE 10 2022 103 683 A1 and in DE 10 2021 200 467 A1.It was the object of the invention to be able to identify interactions in an automated manner.The subject matter of independent claim 1 and of subordinate claims 3-11 achieve this task in each case.According to one aspect, the invention provides a method for generating interaction annotations for a training data set, wherein the training data set comprises sequences and a machine learning model is trainable by means of the generated annotated training data set, the method comprising the steps:• obtaining a sequence of first data, wherein the first data is acquired by at least one first sensor, wherein the first data comprises information on the vehicle environment and / or vehicle interior and / or driving behavior;• obtaining second data, the second data being acquired by a second sensor, the second data comprising at least one interaction of at least one person with the automated vehicle, the first data being synchronized with the second data;• Generating interaction annotations based on the at least one interaction and annotation of the first data with the interaction annotations.Interactions are interactions between at least one person and an automated vehicle. Interactions can consist, for example, in a person opening a vehicle door and entering or exiting the vehicle or loading something into or out of the trunk. However, it can also be a door, for example, which opens automatically when a person approaches this door or actuates a button for automatically opening the door, for example. A self-operatively opening door can be present, for example, if the automated vehicle is, for example, a bus or a shuttle for transporting persons. The person can then be a passenger.An automated vehicle is a vehicle that can perform driving tasks in an automated manner at least in part. Driving tasks can comprise both longitudinal and transverse guidance and also the operation of further actuators, for example the opening and / or closing of a door or, for example, the activation and / or deactivation of a direction indicator or a hazard warning light system. An automated vehicle can be, in particular, a vehicle with automation level 3, 4 and / or 5 according to SAE J3016. A vehicle may be a land-based, water-based, and / or air-based vehicle.The first data that are acquired by the at least one first sensor can comprise, for example, image data and / or point clouds that have been acquired, for example, by an imaging sensor, in particular a camera, a radar sensor and / or lidar sensor.Information about the vehicle interior can also be detected, for example, by a seat occupancy sensor.A first sensor can also be, for example, an acoustic sensor that can detect sounds and / or sounds.However, it can also be data that have been acquired by a sensor that contain information about the driving behavior, in particular about the longitudinal and / or lateral acceleration, for example acquired by an acceleration sensor. However, the driving behavior can also comprise, for example, the activation and / or deactivation of a direction indicator or a hazard warning system.The second data comprises an interaction of at least one person with the automated vehicle and thus sets a trigger in order to identify the first data with an interaction annotation at this point in time. There may be multiple interactions within a sequence.By using the second data as a trigger for identifying the first data, interaction annotations for a training data set for poorly monitored learning can be generated more effectively and quickly than would be possible by manually generated annotations. The second sensor that acquires the second data may be referred to as a reference sensor. The second sensor can be, for example, a door contact switch.In another aspect, the invention provides a computer-implemented method for training a machine learning model for sensing interaction, the method comprising the steps of:• obtaining a training dataset, the training dataset comprising at least one sequence and interaction annotations, wherein the at least one sequence comprises data of an automated vehicle and the data is acquired by at least one sensor, wherein the interaction annotations identify interactions, that is interactions of at least one person with an automated vehicle;• feeding forward the at least one sequence of the training dataset;• obtaining a time domain, the time domain comprising the at least one interaction predicted by the machine learning model;• Comparison of the at least one predicted interaction with the interaction annotations of the training dataset;• backfeed the machine learning model with deviations between the predicted interaction or interactions and the interaction annotations of the training dataset;• Learning the perception of the interaction or interactions in the time domain by gradient-based optimizing weighting factors of the machine learning model.Computer-implemented means that the steps of the method are carried out by a data processing device, for example a computer, a computing system, a computer network, for example a cloud system, hardware of a control unit, or parts thereof.The at least one sequence of the training dataset can comprise image data and / or a trajectory, for example. However, it can also comprise other or further data which contain, for example, information relating to at least one person and / or the automated vehicle.The time domain may include multiple sections. For example, there may be a section before an interaction, a time of an interaction, and a section after an interaction. However, there may also be a plurality of interactions in the time domain and thus in each case sections between the interactions. Machine learning models include artificial neural networks. These can be, for example, recurrent networks, LSTMs or transformers. The training method is a supervised learning method. That is, the machine learning model further optimizes the predicted interactions based on predetermined interaction annotations by optimizing the weighting factors of the machine learning model such that the predicted interactions further conform to the predetermined interaction annotations.According to another aspect, the invention provides a computer-implemented method for training a machine learning model on perception of a trajectory of an automated vehicle before and / or after an interaction, the method comprising the following steps:• obtaining a training dataset, the training dataset comprising at least one sequence and trajectory annotations, wherein the at least one sequence comprises a trajectory of an automated vehicle and the trajectory is detected with the aid of a sensor, wherein the trajectory annotations identify a trajectory before and / or after an interaction of a person with an automated vehicle;• feeding forward the at least one sequence of the training dataset;• obtaining a time range including the trajectory predicted by the machine learning model;• Comparison of the Predicted Trajectory with the Trajectory Annotations of the Training Data Set;• feeding back the machine learning model with deviations between the predicted trajectory and the trajectory annotations of the training dataset;• Learning the perception of the trajectory in the time domain by gradient-based optimizing weighting factors of the machine learning model.Training a machine learning model on a trajectory before interaction may be used, for example, to predict which factors of a trajectory may be related to an interaction. By contrast, predicting a trajectory after an interaction can be used, for example, to predict a driving behavior after an interaction. This method is therefore dependent on interaction annotations. This allows the time range obtained to be divided into a section before and / or after interaction. However, there may also be a plurality of interactions within a sequence. The sections between the interactions can then be divided into sections, for example, wherein the sections identify the trajectory either before or after the interaction. On account of the interactions, there is therefore a technical relationship between the respective methods in the sense of a particular technical feature according to R. 44(1) EPC.According to another aspect, the invention provides a computer-implemented method for training a machine learning model for perception of personal behavior before and / or after interaction, the method comprising the following steps:• obtaining a training dataset, the training dataset comprising at least one sequence and person behavior annotations, wherein the at least one sequence comprises data and the data is acquired by at least one sensor, wherein the person behavior annotations characterize a person behavior before and / or after an interaction of a person with an automated vehicle;• feeding forward the at least one sequence of the training dataset;• obtaining a time domain including the person behavior predicted by the machine learning model;• Comparison of the predicted person behavior with the person behavior annotations of the training dataset;• backfeed the machine learning model with deviations between the predicted person behavior and the person behavior annotations of the training dataset;• Learning the perception of the person behavior in the time domain by gradient-based optimizing weighting factors of the machine learning model.A person behavior can manifest itself both in movement, mimic, gesture and in acoustics. Both before and after a person interacts with an automated vehicle, the person can be located inside or outside the vehicle. Multiple persons may also be involved, where persons may be inside and / or outside of the automated vehicle before and / or after the interaction. The at least one sequence can therefore comprise data of the vehicle environment and / or of the vehicle interior. The data can contain image information and / or sound information and / or movement information, for example. Motion information may also include data sensed by a seat occupancy sensor.This method is dependent on interaction annotations. This allows the time range obtained to be divided into a section before and / or after interaction. However, there may also be a plurality of interactions within a sequence.The sections between the interactions can then be divided into sections, for example, wherein the sections characterize either the person behavior before or after the interaction.On account of the interactions, there is therefore a technical relationship between the respective methods in the sense of a particular technical feature according to R. 44(1) EPC.According to another aspect, the invention provides a computer-implemented method for training a machine learning model on perception of an interaction and / or a trajectory before and / or after an interaction and / or a person behavior before and / or after an interaction, the method comprising the following steps:• obtaining a training dataset, the training dataset comprising at least one sequence and interaction annotations and / or trajectory annotations and / or person behavior annotations, wherein the at least one sequence comprises an interaction and / or a trajectory of an automated vehicle and / or a person behavior and the interaction and / or the trajectory and / or the person behavior is detected with the aid of a sensor;• feeding forward the at least one sequence of the training dataset;• obtaining a time domain comprising the interaction and / or trajectory and / or person behavior predicted by the machine learning model;• Comparison of the at least one predicted interaction and / or trajectory and / or person behavior with the interaction annotations and / or trajectory annotations and / or person behavior annotations of the training dataset;• feeding the machine learning model backward with deviations between the at least one predicted interaction and / or trajectory and / or person behavior and the interaction annotations and / or trajectory annotations and / or person behavior annotations of the training dataset;• Learning the perception of the interaction or interactions and / or trajectory and / or person behavior in the time domain by gradient-based optimizing weighting factors of the machine learning model.Training a machine learning model on perception of an interaction and / or a trajectory before and / or after an interaction and / or a person behavior before and / or after an interaction can be performed on a single machine learning model, for example by means of multi-modal feature representation. Here, the modalities are continuously passed to the machine learning model. The modalities can be imaged, for example, to a shared latent space (a so-called shared latent space). Multimode machine learning is described, for example, by Paul Pu Liang et al. in the publication "Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Responses".According to another aspect, the invention provides a computer-implemented method for sensing interaction, the method comprising the steps of:• obtaining data, wherein the data has been acquired by at least one sensor;• Processing the data by a machine learning model that is trained to recognize interactions;• generating a signal indicative of the interactions determined by the machine learning model;• Outputting the signal to a control device for regulating and / or controlling an automated vehicle.The data may be data collected inside and / or outside the vehicle. The data can comprise information about the vehicle environment and / or vehicle interior and / or driving behavior.The regulation and / or control of an automated vehicle can relate, for example, to the longitudinal and / or transverse guidance of the vehicle, but also, for example, the opening and / or closing of at least one vehicle door or, for example, also the activation and / or deactivation of a direction indicator or a hazard warning light system. However, it can also have an influence on further actuators of an automated vehicle.According to a further aspect, the invention provides a computer-implemented method for perception of a trajectory before and / or after an interaction, the method comprising the steps:• obtaining data, wherein the data has been acquired by at least one sensor;• processing the data by a machine learning model trained to recognize trajectories;• generating a signal indicative of the trajectory determined by the machine learning model;• Outputting the signal to a control device for regulating and / or controlling a longitudinal and / or lateral guidance of an automated vehicle.The perception of a trajectory before and / or after an interaction is dependent on interaction annotations. For example, the type and the time of the interaction influence the trajectory.On account of the interactions, there is therefore a technical relationship between the respective methods in the sense of a particular technical feature according to R. 44(1) EPC.According to a further aspect, the invention provides a computer-implemented method for detecting personal behavior before and / or after an interaction, the method comprising the steps of:• obtaining data, wherein the data has been acquired by at least one sensor;• processing the data by a machine learning model that is trained to recognize person behavior;• generating a signal indicative of the personal behavior determined by the machine learning model;• Outputting the signal to a control device for regulating and / or controlling an automated vehicle.The perception of a person behavior before and / or after an interaction is mutually dependent on interaction annotations. For example, the type and the time of the interaction have an influence on the person behavior on the one hand, and the person behavior can also influence the interaction on the other hand.On account of the interactions, there is therefore a technical relationship between the respective methods in the sense of a particular technical feature according to R. 44(1) EPC.The regulation and / or control of an automated vehicle can relate, for example, to the longitudinal and / or transverse guidance of the vehicle, as already described above, but also, for example, the opening and / or closing of at least one vehicle door or, for example, also the activation and / or deactivation of a direction indicator or a hazard warning light system. However, it can also have an influence on further actuators of an automated vehicle.According to a further aspect, the invention provides a computer program for perception of interaction and / or trajectory before and / or after interaction and / or person behavior before and / or after interaction, the computer program comprising program instructions which cause a computer to execute the steps of a method for perception of interaction and / or trajectory before and / or after interaction and / or person behavior before and / or after interaction when the computer program is loaded or executed on the computer.The instructions of the computer program according to the invention comprise machine instructions, source text or object code written in assembly language, an object-oriented programming language, for example C++, or in a procedural programming language, for example C. The computer program is executed wholly or partly on an integrated circuit or in a remote system comprising a cloud. The integrated circuit may be, for example, a graphics processor (GPU).According to a further aspect, the invention provides a storage medium on which the computer program is stored.The storage medium can be, for example, a nonvolatile, permanent memory or a volatile memory. A storage medium may be, for example, a USB stick, a hard disk, a memory card, or a CD-ROM. However, the storage medium can also be, for example, a cloud in which the computer program is stored.According to a further aspect, the invention provides a data carrier signal which transmits the computer program.The computer program for detecting interaction and / or trajectory before and / or after interaction and / or person behavior before and / or after interaction can be transmitted via a data carrier signal, for example, from a cloud to a control device of an automated vehicle, for example, via a so-called over-the-air update.According to a further aspect, the invention provides an automated vehicle, the automated vehicle comprising at least one control device and at least one sensor, wherein the control device and / or the sensor executes a trained machine learning model and the control device regulates and / or controls the automated vehicle on the basis of a perception of the machine learning model of interaction and / or trajectory and / or person behavior.The sensor may be a sensor that captures data inside and / or outside the vehicle. The data can comprise information about the vehicle environment and / or vehicle interior and / or driving behavior. The regulation and / or control of an automated vehicle can relate, for example, to the longitudinal and / or transverse guidance of the vehicle, as already described above, but also, for example, the opening and / or closing of at least one vehicle door or, for example, also the activation and / or deactivation of a direction indicator or a hazard warning light system. However, it can also have an influence on further actuators of an automated vehicle.Further developments and advantageous embodiments are evident from the dependent claims, the drawings and the description of preferred exemplary embodiments.According to one aspect, the method comprises the following additional steps:• Generating trajectory annotations, wherein the trajectory annotations identify a trajectory of an automated vehicle before and / or after an interaction on the basis of the generated interaction annotations;• Annotation of the first data with the trajectory annotations.Trajectory annotations identify the trajectory of an automated vehicle. The pre-interaction trajectory may be different from the post-interaction trajectory. There may therefore be multiple trajectory annotations.According to the invention, the method comprises the following additional steps:• Generating person behavior annotations, wherein the person behavior annotations identify a person behavior before and / or after an interaction of a person with an automated vehicle on the basis of the generated interaction annotations;• Annotation of the first data with the personal behavior annotations.As already described above, a person behavior can manifest itself, for example, in movement, mimic, gesture and also in acoustics. Both before and after a person interacts with an automated vehicle, the person can be located inside or outside the vehicle. Multiple persons may also be involved, where persons may be inside and / or outside of the automated vehicle before and / or after the interaction.There may be multiple personal behavior annotations.According to the invention, the second sensor detects an opening of a vehicle door.An interaction between at least one person and an automated vehicle can in particular involve opening a vehicle door. The opening of the vehicle door can be effected, for example, by at least one person located inside or outside the vehicle. It is also possible for the vehicle to open the vehicle door automatically. The second sensor can be, for example, a door contact switch.The invention is explained by way of example with reference to the following exemplary embodiments. The following are shown: FIG. 1 shows a flow chart of a method according to the invention, FIG. 2 shows a flow chart of a further method, FIG. 3 shows a flow chart of a further method, FIG. 4 is a flow diagram of a method for training a machine learning model for perception of interaction and for perception of personal behavior before and / or after interaction, FIG. 5 is a flow diagram of a method for training a machine learning model to perceive a trajectory of an automated vehicle before and / or after an interaction, FIG. 6 is a flow diagram of a method for training a machine learning model to detect personal behavior before and / or after interaction, FIG. 7 shows a flow diagram of a method for training a machine learning model on perception of an interaction and / or a trajectory before and / or after an interaction and / or a person behavior before and / or after an interaction, FIG. 8 shows a flow diagram of a method for sensing interaction and person behavior before and / or after an interaction, FIG. 9 shows a flow diagram of a method for perception of a trajectory before and / or after an interaction, FIG. 10 shows a flow diagram of a method for sensing a person behavior before and / or after an interaction, FIG. 11 shows a schematic illustration of an exemplary embodiment of a second sensor, FIG. 12 is a schematic illustration of an embodiment of a computer program, FIG. 13 shows a schematic representation of an embodiment of a storage medium, FIG. 14 shows a schematic illustration of an exemplary embodiment of a data carrier signal, FIG. 15 shows a schematic illustration of an exemplary embodiment of an automated vehicle.In the figures, like reference numerals designate like or functionally like items. In the figures, only the relevant objects are identified by reference numerals.The flow diagram of a method according to the invention shown in FIG. 1 for generating interaction annotations for a training dataset comprises a step I 1 of obtaining a sequence of first data, wherein the first data are acquired by at least one first sensor 3, wherein the first data comprise information about the vehicle environment and / or vehicle interior and / or driving behavior. In addition, the method illustrated in FIG. 1 comprises a step I 2 of obtaining second data, the second data being acquired by a second sensor 2, the second data comprising at least one interaction of at least one person with the automated vehicle 1, the second sensor (2) detecting an opening of a vehicle door, the first data being synchronized with the second data, and a step I 3 of generating interaction annotations on the basis of the at least one interaction and annotation of the first data with the interaction annotations.The flow chart of a further method shown in FIG. 2 comprises, in addition to the steps shown in FIG. 1, a step T 1 of generating trajectory annotations, wherein the trajectory annotations identify a trajectory of an automated vehicle 1 before and / or after an interaction on the basis of the generated interaction annotations. Furthermore, the method illustrated in FIG. 2 comprises a step T 2 of annotation of the first data with the trajectory annotations.The flow chart of a further method shown in FIG. 3 comprises, in addition to the steps shown in FIG. 1, a step P 1 of generating person behavior annotations, wherein the person behavior annotations identify a person behavior before and / or after an interaction of a person with an automated vehicle 1 on the basis of the generated interaction annotations. Furthermore, the method illustrated in FIG. 3 comprises a step P 2 of annotation of the first data with the person behavior annotations.The flow diagram of a method for training a machine learning model for perception of interaction and for perception of a person behavior before and / or after an interaction, shown in FIG. 4, comprises a step MLI 1 of obtaining a training data set, the training data set comprising at least one sequence and interaction annotations and person behavior annotations, wherein the at least one sequence comprises data of an automated vehicle 1 and the data are acquired by at least one sensor 3, wherein the interaction annotations identify interactions, that is interactions of at least one person with an automated vehicle 1, and wherein the person behavior annotations identify a person behavior before and / or after an interaction of a person with an automated vehicle. In addition, the method illustrated in FIG. 4 comprises a step MLI 2 of feeding the at least one sequence of the training dataset forward. Further, the method comprises a step MLI 3 of obtaining a time domain, the time domain comprising the at least one interaction and personal behavior predicted by the machine learning model. In a step MLI 4, a comparison of the at least one predicted interaction with the interaction annotations of the training dataset and a comparison of the predicted person behavior with the person behavior annotations of the training dataset are carried out. In addition, the method includes a step MLI 5 of feeding back the machine learning model with deviations between the predicted interaction or interactions and interaction annotations of the training dataset and with deviations between the predicted person behavior and person behavior annotations of the training dataset, and a step MLI 6 of learning the perception of the interaction or interactions and person behavior in the time domain by gradient-based optimization of weighting factors of the machine learning model.The flow diagram shown in FIG. 5 of a method for training a machine learning model on perception of a trajectory of an automated vehicle 1 before and / or after an interaction comprises a step MLT 1 of obtaining a training dataset, the training dataset comprising at least one sequence and trajectory annotations, wherein the at least one sequence comprises a trajectory of an automated vehicle 1 and the trajectory is detected with the aid of a sensor 3, wherein the trajectory annotations identify a trajectory before and / or after an interaction of a person with an automated vehicle 1. In addition, the method illustrated in FIG. 5 comprises a step MLT 2 of feeding the at least one sequence of the training dataset forward. Further, the method includes a step MLT 3 of obtaining a time range including the trajectory predicted by the machine learning model. In a step MLT 4, the predicted trajectory is compared with the trajectory annotations of the training dataset. In addition, the method includes a step MLT 5 of feeding back the machine learning model with deviations between the predicted trajectory and the trajectory annotations of the training dataset, and a step MLT 6 of learning the perception of the trajectory in the time domain by gradient-based optimizing weighting factors of the machine learning model.The flow diagram of a method for training a machine learning model for perception of a person behavior before and / or after an interaction, shown in FIG. 6, comprises a step MLP 1 of obtaining a training data record, the training data record comprising at least one sequence and person behavior annotations, wherein the at least one sequence comprises data and the data are acquired by at least one sensor 3, wherein the person behavior annotations identify a person behavior before and / or after an interaction of a person with an automated vehicle 1. In addition, the method illustrated in FIG. 6 comprises a step MLP 2 of feeding the at least one sequence of the training dataset forward. Further, the method includes a step MLP 3 of obtaining a time range including the person behavior predicted by the machine learning model. In step MLP 4, a comparison of the predicted person behavior with the person behavior annotations of the training dataset takes place. In addition, the method includes a step MLP 5 of feeding back the machine learning model with deviations between the predicted person behavior and the person behavior annotations of the training dataset, and a step MLP 6 of learning the perception of the person behavior in the time domain by gradient-based optimizing weighting factors of the machine learning model.The flow diagram shown in FIG. 7 of a method for training a machine learning model for perception of an interaction and / or a trajectory before and / or after an interaction and / or a person behavior before and / or after an interaction comprises a step MM 1 of obtaining a training dataset, the training dataset comprising at least one sequence and interaction annotations and / or trajectory annotations and / or person behavior annotations, wherein the at least one sequence comprises an interaction and / or a trajectory of an automated vehicle 1 and / or a person behavior and the interaction and / or the trajectory and / or the person behavior is detected with the aid of a sensor 3. In addition, the method illustrated in FIG. 7 comprises a step MM 2 of feeding the at least one sequence of the training dataset forward. Furthermore, the method comprises a step MM 3 of obtaining a time range comprising the interaction and / or trajectory and / or person behavior predicted by the machine learning model. In step MM 4, the at least one predicted interaction and / or trajectory and / or person behavior is compared with the interaction annotations and / or trajectory annotations and / or person behavior annotations of the training dataset. In addition, the method comprises a step MM 5 of feeding the machine learning model backward with deviations between the at least one predicted interaction and / or trajectory and / or person behavior and the interaction annotations and / or trajectory annotations and / or person behavior annotations of the training dataset and a step MM 6 of learning the perception of the interaction or interactions and / or trajectory and / or person behavior in the time domain by gradient-based optimization of weighting factors of the machine learning model.The flow chart shown in FIG. 8 of a method for sensing interaction and personal behavior before and / or after an interaction comprises a step CI 1 of obtaining data, wherein the data have been acquired by at least one sensor 3. In addition, the method illustrated in FIG. 8 shows a step CI 2 of processing the data by a machine learning model that is trained to recognize interactions and personal behavior. The method further comprises a step CI 3 of generating a signal which points to the interactions determined by means of the machine learning model and to the person behavior determined by means of the machine learning model, and a step CI 4 of outputting the signal to a control device 8 for regulating and / or controlling an automated vehicle 1.The flow diagram of a method for perception of a trajectory before and / or after an interaction, shown in FIG. 9, comprises a step CT 1 of obtaining data, wherein the data have been acquired by at least one sensor 3. In addition, the method illustrated in FIG. 9 includes a step CT 2 of processing the data by a machine learning model that is trained to recognize trajectories. The method further comprises a step CT 3 of generating a signal which points to the trajectory determined by means of the machine learning model, and a step CT 4 of outputting the signal to a control device 8 for regulating and / or controlling a longitudinal and / or lateral guidance of an automated vehicle 1.The flow chart shown in FIG. 10 of a method for detecting a person behavior before and / or after an interaction comprises a step CP 1 of obtaining data, wherein the data have been acquired by at least one sensor 3. In addition, the method illustrated in FIG. 10 includes a step CP 2 of processing the data by a machine learning model trained to recognize personal behavior according to claim 7. The method further comprises a step CP 3 of generating a signal which points to the person behavior determined by means of the machine learning model, and a step CP 4 of outputting the signal to a control device 8 for regulating and / or controlling an automated vehicle 1.The schematic illustration of an exemplary embodiment of a second sensor 2 shown in FIG. 11 shows, by way of example, a door contact switch which is mounted on a door of an automated vehicle 1. The second sensor 2 can be located at the height of a door handle, as illustrated by way of example in FIG. 11. The second sensor can also be integrated into the door handle.The schematic representation of an exemplary embodiment of a computer program 4 shown in FIG. 12 shows, by way of example, a computer program 4 for the perception of interaction and / or trajectory before and / or after an interaction and / or person behavior before and / or after an interaction, wherein the computer program 4 according to FIG. 12 is executed by a computer 5.The schematic representation of an exemplary embodiment of a storage medium 6 shown in FIG. 13 shows, by way of example, a USB stick as storage medium 6, onto which the computer program 4 has been loaded and which can transmit the computer program 4 to a computer 5.The schematic representation of an exemplary embodiment of a data carrier signal 7 shown in FIG. 14 shows, by way of example, a data carrier signal 7 which transmits a computer program 4. According to FIG. 14, the computer program 4 is transmitted, for example, from a cloud by means of the data carrier signal 7.The schematic illustration of an exemplary embodiment of an automated vehicle 1 shown in FIG. 15 shows a passenger car as an example of an automated vehicle 1. according to FIG. 15, the automated vehicle 1 comprises a control unit 8 and a sensor 3. According to FIG. 15, the sensor 3 is located in the vehicle interior in the region behind the windshield. The sensor 3 could therefore be a camera, for example, which can capture data of both the vehicle interior and the vehicle environment.Reference numerals denote reference numeralsI1-I3 Method steps T1-T2 Method steps P1-P2 Method steps MLI1-MLI6 Method steps MLT1-MLT6 Method steps MLP1-MLP6 Method steps MM1-MM6 Method steps CI1-CI4 Method steps CT1-CT4 Method steps CP1-CP4 Method steps 1 Automated vehicle 2 Second sensor 3 Sensor 4 Computer program 5 Computer 6 Storage medium 7 Data carrier signal 8 Control device

Claims

Method for generating interaction annotations for a training dataset, wherein the training dataset comprises sequences and a machine learning model can be trained by means of the generated annotated training dataset, the method comprising the steps: • obtaining a sequence of first data, wherein the first data are acquired by at least one first sensor (3), wherein the first data comprise information on the vehicle environment and / or vehicle interior and / or driving behavior (11); • obtaining second data, wherein the second data are acquired by a second sensor (2), wherein the second data comprise at least one interaction of at least one person with the automated vehicle (1), wherein the second sensor (2) detects an opening of a vehicle door, wherein the first data are synchronized (12) with the second data; • Generating interaction annotations on the basis of the at least one interaction and annotation of the first data with the interaction annotations (I3); • Generating person behavior annotations, wherein the person behavior annotations mark a person behavior before and / or after an interaction of a person with an automated vehicle (1) on the basis of the generated interaction annotations (P1); • Annotation of the first data with the person behavior annotations (P2).Method according to claim 1, wherein the method comprises the following additional steps: • generating trajectory annotations, wherein the trajectory annotations identify (T1) a trajectory of an automated vehicle (1) before and / or after an interaction based on the generated interaction annotations; • annotation the first data with the trajectory annotations (T2).A computer-implemented method for training a machine learning model for perception of interaction and for perception of a person behavior before and / or after an interaction, the method comprising the following steps: • obtaining a training dataset, the training dataset comprising at least one sequence and interaction annotations and person behavior annotations, wherein the at least one sequence comprises data of an automated vehicle (1) and the data are acquired by at least one sensor (3), wherein the interaction annotations characterize interactions, that is interactions of at least one person with an automated vehicle (1), and wherein the person behavior annotations characterize a person behavior before and / or after an interaction of a person with an automated vehicle (MLI1); • feeding forward the at least one sequence of the training dataset (MLI2); • obtaining a time domain, the time domain comprising the at least one interaction and person behavior (MLI3) predicted by the machine learning model; • comparing the at least one predicted interaction with the interaction annotations of the training dataset and comparing the predicted person behavior with the person behavior annotations of the training dataset (MLI4); • feeding back the machine learning model with deviations between the predicted interaction or the predicted interactions and the interaction annotations of the training dataset and with deviations between the predicted person behavior and the person behavior annotations of the training dataset (MLI5); • Learning the perception of the interaction or interactions and the person behavior in the time domain by gradient-based optimizing weighting factors of the machine learning model (MLI6).Computer-implemented method for training a machine learning model on perception of a trajectory of an automated vehicle (1) before and / or after an interaction, the method comprising the following steps: • obtaining a training dataset, the training dataset comprising at least one sequence and trajectory annotations, wherein the at least one sequence comprises a trajectory of an automated vehicle (1) and the trajectory is detected with the aid of a sensor (3), wherein the trajectory annotations identify a trajectory before and / or after an interaction of a person with an automated vehicle (MLT1); • feeding forward the at least one sequence of the training dataset (MLT2); • obtaining a time range comprising the trajectory (MLT3) predicted by the machine learning model; • Comparison of the predicted trajectory with the trajectory annotations of the training dataset (MLT4); • Feedback of the machine learning model with deviations between the predicted trajectory and the trajectory annotations of the training dataset (MLT5); • Learning the perception of the trajectory in the time domain by gradient-based optimizing weighting factors of the machine learning model (MLT6).Computer-implemented method for training a machine learning model on perception of an interaction and / or a trajectory before and / or after an interaction and / or a person behavior before and / or after an interaction, the method comprising the following steps: • obtaining a training dataset, the training dataset comprising at least one sequence and interaction annotations and / or trajectory annotations and / or person behavior annotations, wherein the at least one sequence comprises an interaction and / or a trajectory of an automated vehicle (1) and / or a person behavior and the interaction and / or the trajectory and / or the person behavior is detected (MM1) with the aid of a sensor (3); • feeding forward the at least one sequence of the training dataset (MM2); • obtaining a time range comprising the interaction and / or trajectory and / or person behavior (MM3) predicted by the machine learning model; • comparing the at least one predicted interaction and / or trajectory and / or person behavior with the interaction annotations and / or trajectory annotations and / or person behavior annotations of the training dataset (MM4); • feeding the machine learning model backward with deviations between the at least one predicted interaction and / or trajectory and / or person behavior and the interaction annotations and / or trajectory annotations and / or person behavior annotations of the training dataset (MM5); • Learning the perception of the interaction or interactions and / or trajectory and / or person behaviour in the time domain by gradient-based optimizing weighting factors of the machine learning model (MM6).Computer-implemented method for perception of interaction and of a person behavior before and / or after an interaction, the method comprising the steps: • obtaining data, wherein the data have been acquired (CI1) by at least one sensor (3); • processing the data by a machine learning model trained according to claim 3 to perceive interactions and person behavior (CI2); • generating a signal which points to the interactions determined by means of the machine learning model and to the person behavior determined by means of the machine learning model (CI3); • outputting the signal to a control device (8) for regulating and / or controlling an automated vehicle (1) (CI4).Computer-implemented method for perception of a trajectory before and / or after an interaction, the method comprising the steps: • obtaining data, wherein the data have been acquired (CT1) by at least one sensor (3); • processing the data by a machine learning model trained according to claim 4 to perceive trajectories (CT2); • generating a signal indicative of the trajectory determined by means of the machine learning model (CT3); • outputting the signal to a control device (8) for regulating and / or controlling a longitudinal and / or lateral guidance of an automated vehicle (1) (CT4).Computer program (4) for perception of interaction and / or trajectory before and / or after interaction and / or person behaviour before and / or after interaction, the computer program (4) comprising program instructions which cause a computer (5) to execute the steps of a method according to claim 6 and / or 7 when the computer program (4) is loaded or executed on the computer (5).Storage medium (6) on which the computer program (4) according to claim 8 is stored.A data carrier signal (7) carrying the computer program (4) according to claim 8.Automated vehicle (1) comprising at least one control device (8) and at least one sensor (3), wherein the control device (8) and / or the sensor (3) executes a machine learning model trained according to claim 6 and / or 7 and the control device (8) regulates and / or controls the automated vehicle (1) based on a perception of the machine learning model of interaction and / or trajectory and / or person behavior.

Images