Methods for classifying the behavior of a road user and methods for controlling an ego-vehicle.

Abstract

The invention relates to a computer-implemented method (100) for classifying the traffic behavior of a road user (303) in an environment (305) of an ego-vehicle (301), comprising: - Receiving (101) trajectory data (327) of a movement trajectory (T1) of a road user (303) positioned in the vicinity (305) of an ego vehicle (301); - Selecting (103) at least one latent space representation (335) at least one normal trajectory (T2) from a plurality of latent space representations (335) of a plurality of normal trajectories (T2) in the latent space (333) based on a selection criterion; - Determining (105) a distance (D) of the latent space representation (334) of the motion trajectory (T1) of the road user (303) to the at least one latent space representation (335) selected according to the selection criterion of at least one normal trajectory (T2) in the latent space (333); and - Classifying (107) a traffic behavior of the road user (303) as normal traffic behavior if the distance (D) of the latent space representation (334) of the motion trajectory (T1) of the road user (303) to the latent space representation (335) of the at least one normal trajectory (T2) in the latent space (333) falls below a predetermined limit.

Description

[0001] The invention relates to a method for classifying the traffic behavior of a road user in the environment of an ego vehicle. The invention further relates to a method for controlling an ego vehicle. State of the art

[0002] For controlling vehicles, especially autonomous vehicles, it is essential to accurately assess the future traffic behavior of other road users. For safe vehicle control, it is particularly important to be able to distinguish between normal traffic behavior, in which unpredictable actions are not to be expected, and abnormal traffic behavior, which deviates from normal traffic behavior and in which unpredictable actions must be expected in the future. This allows the driver to adapt the traffic behavior of the controlled vehicle if necessary.

[0003] It is therefore an object of the invention to provide an improved method for classifying a traffic behavior of a road user in an environment of an ego vehicle and an improved method for controlling an ego vehicle.

[0004] This problem is solved by the method for classifying the traffic behavior of a road user in the environment of an ego vehicle and by the method for controlling an ego vehicle of the independent claims. Advantageous embodiments are the subject of the subordinate claims.

[0005] According to one aspect of the invention, a computer-implemented method for classifying a traffic behavior of a road user in an environment of an ego vehicle is provided, comprising: Receiving trajectory data of a movement trajectory of a road user located in the vicinity of an ego vehicle, wherein the trajectory data comprises at least position information of the movement trajectory followed by the road user; Determining a latent space representation of the movement trajectory of the road user in a latent space, taking into account the classification information, wherein the latent space representation of the movement trajectory is given by a dimension-reduced representation of the trajectory data and comprises the classification information; selecting at least one latent space representation of at least one normal trajectory from a plurality of latent space representations of a plurality of normal trajectories in the latent space based on a selection criterion, wherein the normal trajectories are classified as movement trajectories based on normal traffic behavior; Determining a distance of the latent space representation of the movement trajectory of the road user to the at least one latent space representation of at least one normal trajectory in the latent space selected according to the selection criterion; and Classifying a traffic behavior of the road user as normal traffic behavior if the distance between the latent space representation of the movement trajectory of the road user and the latent space representation of the at least one normal trajectory in the latent space falls below a predetermined limit value; or Classifying the traffic behavior of the road user as abnormal traffic behavior if the distance of the latent space representation of the movement trajectory of the road user to the latent space representation of the at least one normal trajectory in the latent space exceeds the predetermined limit value, and outputting classification information.

[0006] This allows the technical advantage of providing an improved method for classifying the traffic behavior of a road user in the environment of an ego vehicle. For this purpose, according to the invention, trajectory data of a road user's movement trajectory is received, and latent space representations are determined for the corresponding trajectory data. The latent space representations are representations of the trajectory data in latent space and describe a dimensionally reduced representation of the trajectory data.

[0007] According to the invention, appropriately trained artificial intelligence calculates corresponding representations in latent space based on the received trajectory data of the movement trajectories and taking into account classification information of a road user. These representations can, for example, be formed in vector representation and represent dimensionally reduced descriptions of the information in the trajectory data.

[0008] The conversion of the trajectory data of the movement trajectories into corresponding latent space representations can thus be interpreted as a coding of the information contained in the trajectory data. The latent space representations contain the information, or most of it, of the corresponding trajectory data, but have a reduced-dimensional representation compared to the representation of the original trajectory data.

[0009] The latent space is seen as the set of all latent space representations of the trajectory data and can be multidimensional according to the dimensionality of the latent space representations.

[0010] The invention is based on the following idea that the appropriately trained artificial intelligence, when converting the trajectory data into corresponding latent space representations, carries out this calculation of the latent space representations according to relevant criteria and thus similar or identical trajectory data lead to similar or identical latent space representations.

[0011] Different trajectory data based on very different movement trajectories, however, will have latent space representations that differ significantly from one another. Latent space representations are defined as vector representations within the meaning of the application. Identical or similar latent space representations are therefore vector representations with identical or only slightly different entries. Slightly different entries result in small differences, while identical entries have a difference of zero.

[0012] Since similar or identical vector representations lead to small or negligible differences, the similar or identical latent space representations will be located very close to each other within the latent space. Strongly different trajectory data, based on very different movement trajectories, in contrast, lead to latent space representations that are located at a greater distance from each other within the latent space.

[0013] Since the latent space representations are preferably formed in vector representations, a simple determination of the distance between the individual latent space representations within the latent space can be used to determine the similarity of the corresponding trajectory data or the corresponding movement trajectories. Similar or identical movement trajectories thus lead to latent space representations with very small distances in the latent space, while strongly different movement trajectories have latent space representations with correspondingly larger distances from each other in the latent space.

[0014] Furthermore, the present invention is based on the idea that movement trajectories based on normal traffic behavior will have strong similarities to each other, while movement trajectories based on abnormal traffic behavior will have strong differences from the movement trajectories of normal traffic behavior.

[0015] Normal traffic behavior will be characterized by compliance with traffic regulations. The vehicle will drive in the prescribed direction of travel in the designated lane of the roadway. No unforeseen maneuvers will be performed. Instead, the vehicle will follow the roadway smoothly, without sudden steering movements and at a constant speed. Furthermore, the vehicle will drive exclusively in the designated lanes and will not cross road markings or, for example, drive on the sidewalk.

[0016] Movement trajectories for such normally driving vehicles will therefore be highly similar to each other, since the courses of the respective movement trajectories are largely defined by the courses of the traffic lanes and the prevailing traffic regulations.

[0017] Vehicles that do not adhere to the course of the road or the prevailing traffic regulations, however, exhibit movement trajectories that will differ significantly from those of normal traffic behavior. These movement trajectories can exhibit any deviations from the behavior of normally moving vehicles described above.

[0018] For example, the trajectories may deviate from the current lanes. Jerky steering movements, serpentine or zigzag movements, or other behaviors may be observed. Significant speed differences or a general disregard for prevailing traffic regulations may be evident. The vehicles in question may deviate from the designated lanes and, for example, drive on sidewalks or in the wrong direction.

[0019] While movement trajectories of normal traffic behavior thus have identical or very similar latent space representations that are arranged very close to each other in latent space, movement trajectories based on abnormal traffic behavior, on the other hand, have latent space representations that are very different from the latent space representations of normal traffic behavior and are arranged correspondingly far from these latent space representations in latent space.

[0020] Based on this idea, the method according to the invention now provides for classifying the traffic behavior of a road user in the environment of the ego vehicle based on the respective movement trajectory as normal traffic behavior or as abnormal traffic behavior by determining the distance of a latent space representation of the trajectory data of the movement trajectory of the road user to known latent space representations of movement trajectories, which are known to be based on normal traffic behavior and which are referred to as normal trajectories in the present application.

[0021] By selecting the latent space distribution based on the selection criteria, classification can be accelerated. The classification of the movement trajectory of the other road user does not need to be performed with respect to all latent space representations of the latent space. Instead, the most relevant latent space representations are selected first, based on the selection criteria. The classification based on the distance determination of the latent space representation of the movement trajectory of the other road user can then be performed exclusively with respect to the selected latent space representations.

[0022] The distance determination of the latent space representations of the movement trajectories of the road users in the environment of the ego vehicle to the latent space representations of the normal trajectories based on normal traffic behavior in the latent space thus enables a simple and precise classification of the traffic behavior of the road user in the environment of the ego vehicle.

[0023] The traffic behavior of the road user in the environment of the ego vehicle is classified as normal traffic behavior if the respective latent space representation of the road user's movement trajectory has a distance to latent space representations of normal trajectories in the latent space that is less than a predetermined threshold.

[0024] However, if the distance between the latent space representation of the road user's movement trajectory and the latent space representations of the normal trajectories in the latent space is greater than the predetermined threshold, the road user's traffic behavior that led to the respective movement trajectory is classified as abnormal traffic behavior.

[0025] Based on the guided classification of the traffic behavior of the road user being examined in the environment of the ego vehicle, the control of the ego vehicle can be adapted to the correspondingly classified behavior of the road user. For example, the ego vehicle can execute predetermined safety functions if abnormal traffic behavior is detected. This, in turn, can improve the control of the ego vehicle and increase its safety.

[0026] The ego vehicle can be a partially automated, highly automated or fully automated vehicle for the purposes of the application.

[0027] Road users may include other vehicles, in particular trucks, cars, buses, motorcycles, or bicycles. A roadway, for the purposes of the application, may be a motorway, a loading lane, an inner-city road, a traffic-calmed road, or a parking lot.

[0028] According to one embodiment, the selection criterion comprises at least one position information of the ego vehicle and / or the other road user.

[0029] This can achieve the technical advantage of selecting relevant latent space distributions by taking the position information into account. Only latent space representations that contain the same or comparable position information as the latent space representation of the movement trajectory are selected for classification. This allows the movement trajectory of the other road user to be compared primarily or exclusively with normal trajectories that contain similar or identical position information. The underlying idea is that, if the position information is identical or similar, the movement trajectory of the other road user and the normal trajectories are located at the same location and follow the same or an adjacent lane.This allows the current course of the roadway to be taken into account when classifying driving behavior. By taking map information from a map display into account, driving behavior can be related to the current course of the roadway traveled by the ego vehicle and / or other road users. The idea here is that driving behavior for a roadway with a first course can represent normal driving behavior, while the same behavior can be classified as abnormal driving behavior if it is on a different course on a different roadway. Taking the course of the roadway traveled into account enables greater precision in the classification of driving behavior.

[0030] According to one embodiment, the position information of the ego vehicle and / or the further road user comprises at least absolute position information in the form of GPS position coordinates.

[0031] This provides the technical advantage that the global position information enables precise position determination of the movement trajectory and the normal trajectories.

[0032] According to one embodiment, the position information of the ego vehicle and / or the further road user comprises at least one relative position information between the ego vehicle and the further road user and at least one position information of a map display depicting the surroundings of the ego vehicle.

[0033] This provides the technical advantage that, by taking the position information from the map display into account, the position information of the other road user can be directly assigned to a specific location. This allows the information from the map display, particularly with regard to the course of the currently traveled roadway, to be directly taken into account. Furthermore, the relative position information is independent of the data format of the original position information. This allows for greater flexibility with regard to data formats. The relative position information is preferably determined taking into account the position of the ego vehicle as the origin of the coordinate system used.

[0034] According to one embodiment, selecting the latent space representation comprises: Performing a search within the latent space based on the absolute position information and / or based on the relative position information and the position information of the map representation; and Determining at least one latent space representation or a predefined number of latent space representations, each of which has position information that is equal to the absolute and / or relative position information and / or the position information of the map representation of the ego vehicle and / or the further road user or has a deviation from the position information of the ego vehicle and / or the further road user that is less than or equal to a predetermined threshold value.

[0035] This allows for the technical advantage of precisely selecting latent space representations with the same or similar position information as the latent space representation of the motion trajectory of the other road user by searching among all latent space representations of the latent space. This improves the selection of the latent space representations to be used for classification.

[0036] According to one embodiment, the position information of the ego vehicle and / or the further road user is determined based on a starting point and / or an end point of the movement trajectory of the ego vehicle and / or the further road user, wherein the starting point and / or the end point are determined with respect to a recording period of trajectory data of the ego vehicle and / or the trajectory data of the further road user.

[0037] This can achieve the technical advantage of enabling precise positioning by taking into account the start and / or end points of the movement trajectories. The recording period can be determined by a predetermined number of consecutive frames from the environment sensors used for environment detection.

[0038] According to one embodiment, the predetermined limit value is designed as a variable limit value, wherein the limit value is designed such that a predetermined percentage of the latent space representations of the latent space distribution have a distance from a center of the latent space distribution that is less than or equal to the predetermined limit value.

[0039] This can achieve the technical advantage of enabling improved classification through the variability of the threshold. The variability of the threshold allows for consideration of differences in the density of the distribution of the various latent space representations in the respective latent space distributions. Not all latent space distributions have the same density of latent space representations. By determining the threshold such that a predetermined percentage of the latent space representations of the latent space distribution have a distance from a center of the respective latent space distribution that is less than or equal to the threshold, the different density of the distributions can be taken into account.

[0040] According to one embodiment, the latent space representations comprise classification information of the road users, wherein the classification information assigns the road users to classes of road users, and wherein the classification is carried out taking into account the classification information by only comparing latent space representations that comprise the same classification information.

[0041] This offers the technical advantage of being able to assign the road user to a clearly defined class of road users, such as vehicles, cyclists, or pedestrians, by taking into account the road user's classification information. This allows the traffic behavior of each road user within the respective class to be assessed. It is assumed that traffic behavior differs for road users in different classes.

[0042] When assessing traffic behavior, only movement trajectories for road users of the same class are compared. If the road user is identified as a vehicle, the respective trajectory data is only compared with the normal trajectories of other vehicles. The same procedure applies if the road user is, for example, a cyclist or pedestrian. Such trajectories are also only compared with the normal trajectories of other cyclists or pedestrians.

[0043] According to one embodiment, the distance is determined by a predetermined distance metric, wherein the distance metric is given by a Euclidean distance function or by a logarithmic likelihood function of the latent space representations within the latent space.

[0044] This allows the technical advantage of enabling a precise distance determination between the latent space representations of the trajectory data of the movement trajectories of the road users in the environment of the ego vehicle and the latent space representations of the known normal trajectories of the known normal traffic behavior within the latent space.

[0045] Since the latent space representations are preferably designed as vector representations, distance determinations between different latent space representations can be carried out with ease. This enables precise distance determination and, based on this, a precise classification of the respective traffic behavior.

[0046] According to one embodiment, the classification is carried out by a classification module, wherein the classification module for determining the latent space representation of the movement trajectory of the road user comprises a correspondingly trained encoder module of an autoencoder architecture, wherein the autoencoder architecture is trained to determine latent space representations of movement trajectories in the latent space and to recognize and classify them as movement trajectories based on normal traffic behavior.

[0047] This provides the technical advantage of enabling precise classification of road users' behavior. For this purpose, an autoencoder architecture is used to learn the latent space representations of the trajectory data, particularly the normal trajectories, during training of the autoencoder architecture. After training is complete, the corresponding encoder module of the autoencoder architecture is configured to generate latent space representations of the trajectory data of movement trajectories.

[0048] As described above, the latent space representations are calculated based on learned criteria, so that similar or identical trajectory data from similar or identical movement trajectories lead to similar or identical latent space representations. Trajectory data from trajectories that deviate significantly from normal trajectories, on the other hand, lead to significantly different or distinct latent space representations. Through appropriate training of the autoencoder architecture, the respective encoder module is thus configured to arrange latent space representations of similar movement trajectories close together in the latent space and to arrange latent space representations of significantly different movement trajectories further apart in the latent space.

[0049] Latent space representations are, within the meaning of the application, arranged close to one another in the latent space if the respective vector representations of the latent space representations have similar or identical entries. However, latent space representations are, within the meaning of the application, arranged at a corresponding distance from one another in the latent space if the respective vector representations of the latent space representations have significantly different entries from one another.

[0050] According to one embodiment, the classification information defines at least the following classes: motor vehicle, truck, bus, motorcycle, tram, cyclist, pedestrian.

[0051] This offers the technical advantage of being able to consider different classes of road users. This allows for precise classification of road user behavior. This takes into account the fact that different road user behaviors are classified as normal or abnormal for different classes of road users.

[0052] According to one embodiment, the method further comprises: Receiving map data of a map display depicting the surroundings of the ego vehicle and / or position / trajectory data of the ego vehicle and / or position / trajectory data of other road users in the surroundings of the ego vehicle, wherein the latent space representations of the trajectory data are determined and / or the distance and / or the behavior of the road user are classified taking into account the map data of the map display and / or the position / trajectory data of the ego vehicle and / or the position / trajectory data of the other road user in the surroundings of the ego vehicle.

[0053] This offers the technical advantage of further improving the precision of assessing traffic behavior. By taking into account the map data and / or the position / trajectory data of the ego vehicle and / or the position / trajectory data of other road users, traffic behavior can be assessed in relation to the course of the roadway being traveled, as well as to the ego vehicle and other road users.

[0054] If, for example, the movement trajectory deviates from the course of the road or the road user does not maintain minimum distances, abnormal driving behavior can be registered.

[0055] According to one embodiment, the method further comprises: Converting the trajectory data into image data of the trajectory data and / or converting the map data into image data of the map data and / or converting the position / trajectory data of the ego vehicle into image data of the position / trajectory data of the ego vehicle and / or converting the position / trajectory data of the other road users into image data of the position / trajectory data of the other road users, wherein the determination of the latent space representation of the movement trajectory and / or the determination of the distance and / or the classification of the behavior of the road user is carried out based on the image data of the trajectory data and / or on the image data of the map data and / or the image data of the position / trajectory data of the ego vehicle and / or the image data of the position / trajectory data of the other road users.

[0056] This can achieve the technical advantage that the image data of the trajectory data and / or the image data of the map data of the map display enable a more precise determination of the latent space representations of the trajectory data and / or the map data. This, in turn, enables a more precise classification of the traffic behavior of the road users of the ego vehicle.

[0057] According to one embodiment, the classification module further comprises a connection network, wherein the connection network is configured to combine a plurality of image data of trajectory data and / or a plurality of image data of the map data and / or a plurality of image data of the position / trajectory data of the ego vehicle and / or a plurality of image data of the position / trajectory data of the other road users into input data for the encoder module in vector representation, and / or wherein the classification module further comprises a label network, wherein the label network is configured to determine labeling information for sampled latent space representations based on the image data of the map data and / or the image data of the position / trajectory data of the ego vehicle and / or the image data of the position / trajectory data of the other road users,and wherein the connection network and / or the label network are designed as appropriately trained neural networks.,

[0058] This can achieve the technical advantage of enabling a further improvement in the classification of road user behavior. To this end, a connection network of the classification module first combines the majority of image data from the trajectory data and / or the image data from the map data and / or the image data from the position / trajectory data of the ego vehicle and / or the image data from the position / trajectory data of the other road users into input data for the encoder module.

[0059] By combining the image data of the trajectory data and the map data and the position / trajectory data of the ego vehicle and the position / trajectory data of the other road users, the position information of the trajectory data can be compared with the position information of the map data and the position data of the ego vehicle or the other road users, thus achieving a more precise position determination or a more precise comparison of the courses of the movement trajectories with the courses of the roadway being traveled and the movement trajectories of the ego vehicle or the other road users.

[0060] By taking the ego vehicle and / or other road users into account, the assessment of traffic behavior can be carried out taking into account the ego vehicle and other road users.

[0061] In particular, the trajectory data and the map data and the position / trajectory data of the ego vehicle and the position / trajectory data of the other road users can be processed jointly by the connection network into the common input data for the encoder module, thereby achieving an additional increase in the precision of the latent space representations of the trajectory data and the map data.

[0062] The classification module also includes a label network configured to convert the image data of the map data into corresponding label information. This can later be used as label information of the sampled latent space representations as input data for the decoder module. The autoencoder architecture can thus be used as a conditional variational autoencoder.

[0063] According to one embodiment, the autoencoder architecture is designed as a variational autoencoder VAE or as a conditional variational autoencoder VAE, and wherein the latent space representations of the movement trajectories are designed as statistical distributions in the latent space and each have an expectation value µ and a variance σ 2 are characterized.

[0064] This allows for the technical advantage of introducing an uncertainty aspect into the training of the autoencoder architecture by designing the autoencoder architecture as a variational autoencoder (VAE) and, in particular, by designing the latent space representations as statistical distributions in the latent space. This avoids the problem of overfitting during the training of the autoencoder architecture. This enables an improved training process and a correspondingly better trained encoder module.

[0065] According to one embodiment, the distributions in the latent space are formed as Gaussian normal distributions.

[0066] This provides the technical advantage of achieving the simplest and most manageable statistical distribution of latent space representations. The Gaussian normal distribution facilitates the handling of the statistical distributions of latent space representations.

[0067] According to one embodiment, the training of the autoencoder architecture is carried out on trajectory data of a plurality of journeys of a vehicle or a plurality of vehicles along a roadway traveled by the ego vehicle, wherein map information of a map display depicting the respective roadway traveled and / or position / trajectory data of the ego vehicle and / or position / trajectory data of the other road users and / or information regarding road surface conditions and / or information regarding prevailing traffic regulations and / or information regarding prevailing traffic volumes and / or information regarding prevailing weather conditions and / or time of day and / or day of week information are also taken into account for training.

[0068] This offers the technical advantage of enabling precise training of the autoencoder architecture. For this purpose, trajectory data recorded from multiple journeys of multiple vehicles, or at least one vehicle, along different roadways is considered as training data. For example, to train the autoencoder architecture, only trajectory data from movement trajectories based on the normal traffic behavior of the respective vehicles can be considered.

[0069] The corresponding autoencoder architecture is thus trained during training to generate the corresponding latent space representations of the trajectory data of the normal trajectories and, due to the high similarity of the normal trajectories, to arrange them in close proximity to one another in the latent space. To increase the precision of calculating the latent space representations of the trajectory data of the normal trajectories, information from a map display, road surface conditions, prevailing traffic regulations, traffic volume, weather conditions, or time-of-day and / or day-of-week information can also be considered for training, each of which further describes the conditions under which the trajectory data was recorded during the majority of vehicle journeys.

[0070] Furthermore, position / trajectory information of the ego vehicle and / or position / trajectory information of other road users can be taken into account. This additional information allows for more detailed specification of the traffic behavior of the vehicles that can be classified as normal. The consideration of the various information categories is based on the idea that the respective traffic behavior can vary depending on traffic volume, weather conditions, road surfaces, time of day, or day of the week.

[0071] Taking this into account thus enables a precise definition of normal traffic behavior and enables the autoencoder architecture to more accurately classify and create the respective latent space representations of the recorded trajectory data. After training is complete, the encoder module of the autoencoder architecture is better equipped to generate a corresponding latent space representation, which enables the corresponding classification module to better classify the traffic behavior of the road users in the vicinity of the ego vehicle.

[0072] According to one embodiment, normal traffic behavior of the road user includes: Driving on a designated road in accordance with prevailing traffic regulations Lane of the roadway in use; and / or Driving in accordance with prevailing speed limits; Consequences of a road course; and / or Driving in a direction of travel (DIR) designated by prevailing traffic regulations for the respective carriageway and / or lane; and / or Maintaining a predefined minimum distance from other road users or the ego vehicle; and / or Driving on a tramway track if the road user is a tramway; and / or Riding on a designated cycle path if the road user is a cyclist; and / or Walking / standing on a sidewalk designated for pedestrians, if the road user is a pedestrian; and where abnormal road user behavior includes: Not following the course of the road; and / or Driving in a serpentine direction that deviates from a straight direction; and / or Driving in a direction opposite to that indicated by the prevailing traffic regulations; and / or Driving at excessive speed; and / or Driving in a lane not designated according to the prevailing traffic regulations; and / or Failure to maintain the predefined minimum distance to other road users and / or the ego vehicle; and / or Driving in a lane not designated for bicycles if the road user is a cyclist; and / or Walking / standing in a lane designated for vehicles if the road user is a pedestrian.

[0073] This provides the technical advantage of enabling a precise classification of the traffic behavior of other road users.

[0074] According to one embodiment, the received trajectory data of the further road user in the environment of the ego vehicle are based on environment sensor data of at least one environment sensor of the ego vehicle, wherein the environment sensor data comprise camera data, LiDAR data, RADAR data and / or acoustic data.

[0075] This provides the technical advantage of enabling precise determination of the trajectory data of the movement trajectories of the road users in the vicinity of the ego vehicle. This allows for precise classification of the traffic behavior of the respective road users.

[0076] According to a further aspect, a method for controlling an ego vehicle is provided, comprising: Carrying out a classification of a traffic behavior of a road user in the environment of an ego vehicle based on trajectory data of the road user according to the method for classifying a traffic behavior of a road user of an ego vehicle according to one of the preceding embodiments; and Controlling the ego vehicle taking into account the classification of the traffic behavior of the other road user.

[0077] This allows the technical advantage of providing an improved method for controlling an ego vehicle. By implementing the method for classifying the traffic behavior of a road user of an ego vehicle, an improved classification with the aforementioned technical advantages can be achieved.

[0078] According to one embodiment, controlling the ego vehicle comprises: executing a safety maneuver if the traffic behavior of the other road user is classified as abnormal traffic behavior.

[0079] This offers the technical advantage of enabling an appropriate response to the classified traffic behavior of the road user, allowing the ego vehicle to drive safely. By executing the safety maneuver, the system can react to abnormal traffic behavior of other road users, ensuring the safety of the ego vehicle.

[0080] According to one embodiment, the safety maneuver includes: Reducing the speed of the ego vehicle; and / or Increasing the distance between the ego vehicle and other road users and / or Changing a lane of the roadway occupied by the ego vehicle.

[0081] This can achieve the technical advantage of executing effective safety maneuvers that can ensure the safety of the ego vehicle. For the purposes of the application, the safety of the ego vehicle refers to collision-free travel of the ego vehicle without colliding with other road users. The safety maneuver to be executed can be selected, in particular, taking into account the classified abnormal behavior of the other road user.

[0082] According to a further aspect, a computing unit is provided which is configured to carry out the method for classifying a traffic behavior of a road user in the environment of an ego vehicle according to one of the preceding embodiments and / or the method for controlling an ego vehicle according to one of the preceding embodiments.

[0083] According to a further aspect, a computer program product comprising instructions is provided which, when the program is executed by a data processing unit, cause the data processing unit to execute the method for classifying a traffic behavior of a road user in the environment of an ego vehicle according to one of the preceding embodiments and / or the method for controlling an ego vehicle according to one of the preceding embodiments.

[0084] Embodiments of the invention are explained with reference to the following drawings. The drawings show: Fig. 1 a schematic representation of a system for classifying a traffic behavior of a road user in the environment of an ego vehicle; Fig. 2 a graphical representation of a classification of traffic behavior; Fig. 3 schematic representations of two driving situations of an ego vehicle; Fig. 4 is a graphical representation of an autoencoder architecture according to one embodiment; Fig. 5 a flowchart of a method for trajectory planning for an ego vehicle; Fig. 6 shows another flowchart of the trajectory planning method for an ego vehicle; Fig. 7 is a flowchart of a method for controlling an ego vehicle; and Fig. 8 a schematic representation of a computer program product.

[0085] Fig. 1 shows a schematic representation of a system 300 for classifying a traffic behavior of a road user 303 in the environment 305 of an ego vehicle 301.

[0086] Fig. 1 shows a traffic situation with an ego vehicle 301 traveling on a roadway 302. Another road user 303, in the embodiment shown as another vehicle, is located in the environment 305 of the ego vehicle 301. The ego vehicle 301 has a computing unit 307 with a classification module 309. The classification module 309 is configured to execute the inventive method for classifying the traffic behavior of another road user 303 in the environment 305 of an ego vehicle 301.

[0087] To classify the traffic behavior of road user 303, the computing unit 307 or the classification module 309 first receives trajectory data 327. The trajectory data 327 describes a movement trajectory of road user 303 and includes at least position information of road user 303 at various points in time. The trajectory data 327 can additionally include speed information of road user 303.

[0088] The ego vehicle 301 further comprises at least one environment sensor 313, by means of which the environment 305 of the ego vehicle 301 can be viewed. The trajectory data 327 and the classification information 357 can thus be provided by environment sensor data of the environment sensor 313. The movement trajectory road user class executed by the road user 303 can therefore be detected by the environment sensors 313 of the ego vehicle 301 and determined, for example, by environment detection. The environment sensor data can include LiDAR data, radar data, camera data, and / or acoustic data from corresponding acoustic sensors.

[0089] To classify traffic behavior, the classification module 309 is configured to calculate latent space representations 325 for the received trajectory data 327 of the road user 303. The latent space representations 325 describe dimension-reduced representations of the trajectory data 327. The latent space representations 325 thus represent representations of the corresponding trajectory data 327 in a latent space 333 and can be embodied, for example, as vector representations of the trajectory data 327.

[0090] In the embodiment shown, the classification module 309 comprises a decoder module 317 of an autoencoder architecture 315 shown in simplified form in graphic a).

[0091] In addition to the encoder module 317, the autoencoder architecture 315 comprises a corresponding decoder module 319. The autoencoder architecture 315 is configured to calculate latent space representations 325 in the latent space corresponding to the input data 321. Since the latent space representations 325 of the input data 321 at least largely comprise the information of the input data 321, but are reduced in their dimensionality compared to the input data 321, the latent space representations 325 represent an encoding of the information of the input data 321.

[0092] The decoder module 319 of the autoencoder architecture 315 is further configured to reproduce the input data 321 based on the latent space representations 325 of the input data 321 and to generate correspondingly reproduced input data 323.

[0093] By appropriate training of the autoencoder architecture 315, corresponding latent space representations 325 can be created for given input data 321.

[0094] According to the invention, the autoencoder architecture 315 is trained based on trajectory data 327. The trajectory data 327 is based on a plurality of journeys of a vehicle or a plurality of vehicles and describes the movement trajectories executed by the vehicles.

[0095] Furthermore, classification information 357 can be considered for classification. Classification information 357 assigns road user 303 to a class of road users. Road user 303 can be identified, for example, as a motor vehicle, bus, truck, tram, motorcyclist, cyclist, pedestrian, or the like.

[0096] For this purpose, the classification information 357 can be integrated into the latent space representations 325. The classification information 357 can also be taken into account in the training of the autoencoder architecture 315.

[0097] The training of the autoencoder architecture 315 can further be carried out taking into account map data 329, which includes map information of the roadways traveled by the respective vehicles to record the trajectory data 327. Furthermore, information regarding the road surface condition, traffic volume, weather conditions, times of day or days of the week, as well as prevailing traffic regulations that were present during the journeys to record the trajectory data 327 can be taken into account for training. In addition, position / trajectory data 353 of the ego vehicle 301, position / trajectory data 355 of other road users, and the classification information 357 can be taken into account in the training.

[0098] The corresponding trajectory data 327, map data 329, or data of the aforementioned information 331 can be taken into account in the shown input data 321 of the autoencoder architecture 315. Furthermore, position / trajectory data 353 of the ego vehicle 201 and / or position / trajectory data 355 of other road users and the classification information 357 can be taken into account.

[0099] After appropriate training of the autoencoder architecture 315, the respective encoder module 317 is configured to calculate corresponding latent space representations 325 in the latent space for the received trajectory data 327 of the movement trajectory executed by the road user 303.

[0100] During the training of the autoencoder architecture 315, trajectory data 327 of movement trajectories based on normal traffic behavior of the respective vehicles or road users are primarily used. The trajectory data 327 are labeled according to the corresponding class of the respective road users. Normal traffic behavior is characterized by the fact that prevailing traffic regulations are taken into account, that the respective vehicle follows the course of the roadway evenly, that no jerky or unpredictable steering movements or maneuvers are performed, and that, for example, a uniform speed profile is present. Normal traffic behavior can also be characterized by other features not mentioned here. Normal traffic behavior usually corresponds to the average traffic behavior of road users.

[0101] Furthermore, it can be considered that the road user is riding on the designated cycle path if the road user is a cyclist. Furthermore, it can be considered that the road user is walking on the sidewalk if the road user is a pedestrian.

[0102] Normal traffic behavior can thus be adapted to the respective class of road user. Normal traffic behavior can be adapted to the prevailing weather conditions, the prevailing traffic volume, the time of day or day of the week, as well as the respective road conditions and the prevailing traffic regulations.

[0103] The characteristics of normal traffic behavior mentioned here are merely examples. Depending on the traffic situation, normal traffic behavior may include more or fewer of the mentioned characteristics. This is not intended to limit the invention.

[0104] Movement trajectories of normal traffic behavior, referred to as normal trajectories in the application, exhibit great similarities to one another and differ only slightly. The corresponding movement trajectories largely share common paths that correspond to the paths of the respective roadway and are also similar to one another in terms of steering movements, maneuvers performed, and the respective vehicle speed.

[0105] During the training of the autoencoder architecture 315, the encoder module 317 is trained to convert the trajectory data 327 into corresponding latent space representations 325. According to the invention, the latent space representations generated in this way each comprise a selection criterion as additional information. The selection criterion can, for example, comprise position information. Via the position information, information regarding the locations at which the movement trajectories or normal trajectories of the respective latent space representations were recorded is integrated into the latent space representations. Thus, using the position information of the selection criterion, latent space representations can be identified with respect to the respective locations at which the movement or normal trajectories were recorded.Location refers to a geographical location along which the vehicle or the respective road user drove or moved during the recording of the trajectory data.

[0106] The latent space representations 325 can further comprise the classification information 357. The encoder module 317 is trained to convert similar or identical trajectory data 327 into similar or identical latent space representations 325. According to the invention, similar or identical latent space representations 325 have only small distances from one another in the latent space and are arranged in common regions in the latent space. Similar or identical latent space representations 325 describe vector representations whose entries are equal or have only small deviations from one another. The similarity can be defined using predefined threshold values. Latent space representations with distances from one another in the latent space that are smaller than the predefined threshold value are then defined as similar or identical.

[0107] Movement trajectories, on the other hand, which are based on abnormal traffic behavior, which includes, for example, non-compliance with the prevailing traffic regulations, non-compliance with the course of the roadway being traveled on, or the presence of jerky steering movements, uneven speed profiles, or, for example, zigzag-shaped or serpentine driving, are converted by the appropriately trained encoder module 317 into corresponding latent space representations 325, which have a relatively large distance in the latent space to the latent space representations 325 of the normal trajectories.

[0108] In order to classify the movement trajectories of the road users 303 arranged in the environment 305 of the ego vehicle 301, the classification module 309 is configured to determine distances between latent space representations 325 of the received trajectory data 327 of the movement trajectories driven by the road users 303 in the latent space to latent space representations 325 of normal trajectories that were learned during the training process of the autoencoder architecture 315 by the autoencoder architecture 315 and in particular by the encoder module 317.

[0109] For this purpose, the classification module 309 comprises, in addition to the encoder module 317, a distance module 318. The distance module 318 is configured to determine distances D of the latent space representations 325 of the received trajectory data 327 of the movement trajectories of the road users 303 to the learned latent space representations 325 of the normal trajectories.

[0110] The distances D can be determined by the distance module 318 according to a predetermined distance metric within the latent space. The distance metric can be given, for example, by a Euclidean distance function or by a logarithmic likelihood function.

[0111] The classification module 309 further comprises a selection module 320. According to the invention, prior to the distance determination by the distance module 318, certain latent space representations 325 are selected from among all latent space representations 325 of the normal trajectories in the latent space 333. The selection module 320 effects this selection taking into account the selection criterion.

[0112] As described above, the selection criterion may, for example, include position information of the ego vehicle 301 and / or the other road user 303.

[0113] The position information of the ego vehicle 301 and / or the further road user 303 can each comprise absolute position information in the form of GPS position coordinates.

[0114] Alternatively or additionally, the position information of the ego vehicle 301 and / or the further road user 303 can each comprise relative position information between the ego vehicle 301 and the further road user 303 and position information of the map display depicting the surroundings 305 of the ego vehicle 301.

[0115] Using the position information, the selection module 320 can select the latent space representations 325 whose normal trajectories have passed through the same positions as the movement trajectory of the other road user 303.

[0116] The movement trajectory of the further road user 303 is thus classified with respect to the normal trajectories that were recorded at the same location, i.e. on the same roadway or in the same roadway area, as the movement trajectory of the further road user 303.

[0117] The similarity of the locations can, in turn, be determined using correspondingly predefined thresholds. Motion trajectories and / or normal trajectories with latent space representations containing position information, each with differences less than or equal to the respective threshold, are defined as being recorded at the same location.

[0118] The position information can be determined taking into account the starting points and / or end points of the normal trajectories or the movement trajectory. The starting points and end points can be determined with respect to the recording periods of the movement trajectory or the normal trajectories.

[0119] According to the invention, the position information is integrated into the latent space representations 325 of the normal trajectories and the movement trajectories of the other road user 303.

[0120] By means of the aforementioned functions, the distance module 318 can thus determine distances between the latent space representations 325 of the received trajectory data 327 of the movement trajectories of the road users 303 and the latent space representations 325 of the normal trajectories based on normal traffic behavior, which were learned in the training process.

[0121] Based on the determined distances D, the classification module 309 classifies the traffic behavior of the road users 303 located in the environment 305. A traffic behavior of a road user 303 is classified as normal traffic behavior if the respective latent space representations 325 of the corresponding movement trajectories of the road user 303 have a distance D to latent space representations 325 of the normal trajectories in the latent space that is less than a predetermined threshold.

[0122] However, if the latent space representations 325 of the movement trajectories executed by the respective road user 303 have distances D to the latent space representations 325 of the normal trajectories in the latent space that are greater than or equal to the predetermined limit value, the corresponding traffic behavior of the road user 303 is classified as an abnormal traffic behavior.

[0123] According to the invention, classification information is subsequently output in which the classification of the traffic behavior of the road user 303 arranged in the environment 305 is described.

[0124] Based on the classification information, the control of the ego vehicle 301 can be adjusted. For example, a safety maneuver can be executed or initiated if the traffic behavior of the road user 303 is classified as abnormal traffic behavior. The corresponding safety maneuver can, for example, include a reduction in the speed of the ego vehicle 301 and / or an increase in the distance between the ego vehicle 301 and the respective road user 303.

[0125] Alternatively or additionally, the safety maneuver may include a lane change from a busy lane 306 to a non-busy lane 304. Alternatively or additionally, further maneuvers may be performed by means of which the traffic behavior of the road user 303 located in the surrounding area 305, which is classified as abnormal traffic behavior, is taken into account accordingly.

[0126] The autoencoder architecture 315 and in particular the encoder module 309 can be designed according to architectures for autoencoders known from the prior art.

[0127] According to one embodiment, the autoencoder architecture 315 is designed as a variational autoencoder VAE or a conditional variational autoencoder CVAE. The correspondingly generated latent space representations 325 are designed as statistical distributions in the latent space and are each characterized by an expected value µ and a variance σ 2 The statistical distributions of the latent space representations 325 can, for example, be Gaussian normal distributions.

[0128] The training process of the autoencoder architecture 315 can in particular be designed as a supervised learning process.

[0129] According to one embodiment, the training of the autoencoder architecture 315 can be carried out based on image data of the trajectory data 327 and / or on image data of the map data 329 and / or image data of the position / trajectory data 353 of the ego vehicle 301 and / or image data of the position / trajectory data 355 of other road users.

[0130] For this purpose, the autoencoder architecture includes 315 additional and Fig. 1 not shown components that are configured to convert the image data into the input data 317 in vector representation. For a detailed description of the autoencoder architecture 315, please refer to the description of the Fig. 4 and Fig. 5.

[0131] Analogously, after successfully completing training, the classification module 309 can be executed to classify the behavior of the road user 303, in addition to the trajectory data 327 of the road user 303, taking into account map data 329. In particular, the classification module 309 can classify the behavior of the road user 303 based on the image data of the trajectory data 327 and / or the map data 329. For this purpose, the classification module 309 can include the additional components of the autoencoder architecture 315 for calculating the input data 317 in vector representation.

[0132] Fig. Figure 2 shows a graphical representation of a classification of traffic behavior.

[0133] In Fig. 2 shows a graphical and highly simplified representation of the latent space 333 and the respective latent space representations 335 of the normal trajectories T2 as well as the latent space representations 334 of the movement trajectories T1 of the road users 303 arranged in the environment 305 of the ego vehicle 301.

[0134] The latent space representations 335 of the normal trajectories T1 are arranged in latent space distributions 337. The multiple latent space representations 335 combined into a latent space distribution 337 exhibit a high degree of similarity and describe normal trajectories T2, which also exhibit a high degree of similarity. Since the normal trajectories T2 describe normal driving behavior, a high degree of similarity between the normal trajectories T2 is to be expected, since the paths of the normal trajectories T2 are strongly restricted by the paths of the respective lanes traveled and by the prevailing traffic regulations.

[0135] According to the invention, the latent space representations 334, 335 are provided with the selection criteria. As mentioned, the selection criteria can include the position information. The position information defines the positions or locations at which the respective normal trajectories T2 and the movement trajectory T1 were recorded.

[0136] Preferably, the latent space representations 334 of the normal trajectories T2 are arranged in a combined manner with respect to the respective position information to form latent space distributions 337. The latent space distributions 337 thus comprise latent space representations 335 of normal trajectories T2 that were recorded at identical or similar locations.

[0137] During the training process of the autoencoder architecture 315, the corresponding encoder module 317 is trained to calculate the latent space representations 335 of the trajectory data 327 used in the training process. The autoencoder architecture 315, and in particular the encoder module 317, learn to convert the standard trajectories T2 representing normal traffic behavior, which are highly similar to one another despite being executed by different vehicles, into likewise highly similar latent space representations 335, which are arranged at short distances from one another within the latent space 333.

[0138] After successful training, the encoder module 317 is thus configured to calculate corresponding latent space representations 334 in the latent space 333 for the observed movement trajectories T1 of the detected road users 303 and in particular for the corresponding trajectory data 327. If the movement trajectories T1 of the road users 303 are based on normal traffic behavior, the corresponding latent space representations 334 will exhibit a high degree of similarity to the latent space representations 335 of the normal trajectories T2 and will be arranged in the latent space 333 at a short distance from the respective latent space representations 334 of the normal trajectories T2.

[0139] If, however, the movement trajectories T1 of the road users 303 are based on abnormal traffic behavior, the corresponding movement trajectories T1 exhibit strong differences from the normal trajectories T2 and the correspondingly generated latent space representations 334 of the movement trajectories T1 exhibit a correspondingly large distance D from the latent space representations 335 of the normal trajectories T2 in the latent space 333.

[0140] In Fig. 2 shows a situation of abnormal traffic behavior of a road user 303. The corresponding movement trajectory T1 shows a large difference from the normal trajectories T2, which Fig. 2 is represented by the serpentine course of the movement trajectory T1 compared to the straight curve of the normal trajectory T2. The correspondingly generated latent space representation 334 of the movement trajectory T1 thus has a relatively large distance D from the latent space representations 335 of the normal trajectories T2.

[0141] In Fig. 2 shows three latent space distributions 337 of latent space representations 335 for different normal trajectories T2 in the latent space 333.

[0142] In the embodiment shown, the latent space representations 335 of the three different latent space distributions 337 have different position information. The respective normal trajectories T2 were recorded at different locations accordingly.

[0143] This is merely an example. In reality, a plurality of different trajectories T2 can be determined, all of which are based on normal traffic behavior. They differ from one another, but have a large number of similarities due to the underlying normal traffic behavior and are thus arranged in a comparatively narrowly defined area within latent space 333.

[0144] To classify behavior based on the distance determination of the latent space representation 334 of the motion trajectory T1 to the latent space representations 335 of the normal trajectories T2, the latent space 333 is first searched by the selection module 320. The search is performed taking into account the selection criterion. For this purpose, the selection criterion is integrated into the latent space representations 334, 335 as additional information.

[0145] Using the position information of the selection criterion, the corresponding movement trajectories T1 and normal trajectories T2 are pre-classified to the locations where the respective trajectory data were recorded. The selection module 320 thus preferentially selects for classification the latent space representations 334, 335 of the normal trajectories T2 that were recorded at the same locations as the movement trajectory T1 of the other road user 303.

[0146] The position information can be absolute position information in the form of GPS coordinates or relative position information. The relative position information describes the positioning of the other road user 303 relative to the ego vehicle 301.

[0147] In addition to the position information, the selection criteria can contain additional information. This additional information can be information from the map display, such as information regarding the road surface or the presence of construction sites.

[0148] The distance determination by the distance module 318 is subsequently carried out between the latent space representation 334 of the movement trajectory T1 of the further road user 303 and the latent space representations 335 of the normal trajectories T2 selected by the selection module 320 according to the selection criterion.

[0149] In Fig. In Figure 2, the distances between the latent space representations 334, 335 are not to scale, so that the impression may arise that the distance D between the latent space representation 334 and the latent space representations 335 is smaller than the distances between the latent space representations 335. This is unintentional and is due to the representation.

[0150] According to the invention, a predetermined threshold value is defined for classifying the traffic behavior of road users 303, and in particular for determining the distance D between the latent space representations 334 of the movement trajectories T1 and the latent space representations 335 of the normal trajectories T2. This threshold value can be individually adjusted for different driving situations depending on various criteria, for example, the degree of similarity of the normal trajectories T2.

[0151] According to one embodiment, the limit value is configured as a variable limit value. The limit value is thus not applied with a uniform value to all latent space distributions. This means that the respective limit value is adapted to the respective latent space distributions 337. Different latent space distributions 337 can have different density distributions of the respective latent space representations 334 summarized in the latent space distributions 337. To account for this, the limit value is configured such that, for a latent space distribution 337, a predetermined percentage of the latent space representations 334 of the latent space distributions 337 have a distance from a center of the latent space distributions 337 that is less than or equal to the limit value. The limit value is thus defined as a distance that can be different for different latent space distributions 337.

[0152] For driving situations in which the normal trajectories T2 differ greatly from one another due to the prevailing driving situation, even though they are all based on normal traffic behavior, a correspondingly larger threshold value can be selected for the distance determination. For driving situations in which the normal trajectories T2 differ only slightly from one another and the corresponding latent space representations 335 of the individual normal trajectories T2 are arranged in the latent space 333 in a correspondingly smaller area with smaller distances from one another, the predetermined threshold value can be correspondingly assigned a lower numerical value, so that even smaller distances between the latent space representation 334 of the movement trajectories T1 of the road users 303 and the latent space representations 335 of the normal trajectories T2 already lead to the classification of normal traffic behavior of the respective road users 303.

[0153] In the embodiment shown, the autoencoder architecture 315 is implemented as a variational autoencoder (VAE). The latent space representations can be implemented as statistical distributions. The statistical distributions are defined by expected values ​​µ and variances σ. 2 marked and are in Fig. 2 by the circular structures. The individual latent space representations 335 thus exhibit corresponding planar formations in the latent space 333 according to the statistical distributions, so that the latent space 333 is formed by a continuous occupancy of the latent space representations.

[0154] As already mentioned Fig. 1, the classification module 309 comprises, in addition to the encoder module 317, the previously described distance module 318. The distance module 318 is configured to determine the distances D between the latent space representations 334 of the movement trajectories T1 of the road users 303 located in the environment 305 of the ego vehicle 301, determined via the received trajectory data 327, and the latent space representations 335 of the normal trajectories T2 based on normal traffic behavior, learned in the training process of the autoencoder architecture 315. This can be done, as already described, by executing a corresponding distance metric, which can be implemented, for example, as a Euclidean distance function or a logarithmic likelihood function.

[0155] According to one embodiment, the Fig. The latent space representations 334, 335 shown in Figure 2 each include classification information in addition to the trajectory data. The latent space representations 334, 335 are thus limited to the respective class of the respective road user 303. This ensures that only movement trajectories originating from road users of the same class are compared.

[0156] If the road user 303 is recognized as a motor vehicle, the respective movement trajectories T1 are only compared with normal trajectories T2 of other motor vehicles.

[0157] Fig. 3 shows schematic representations of two driving situations of an ego vehicle 301.

[0158] The two graphs a) and b) of the Fig. 3 show two different driving situations with an ego vehicle 301 and another road user 303. In both driving situations, both the ego vehicle 301 and the other road user 303 are driving in a lane 302. The ego vehicle 301 and the other road user 303 are driving in opposite directions in the driving situations shown.

[0159] In graphic a), the other road user 303 exhibits normal traffic behavior. This is characterized by the relatively straight movement trajectory T1 of road user 303, the course of which largely follows the course of the traveled roadway 302. The ego vehicle 301 also exhibits normal traffic behavior, which in turn is represented by the relatively straight movement trajectory T3, which in turn follows the course of the traveled roadway 302.

[0160] In graph b), however, road user 303 exhibits abnormal traffic behavior. This is illustrated by the more or less serpentine movement trajectory T2 of road user 303.

[0161] Fig. 4 shows a graphical representation of an autoencoder architecture 315 according to one embodiment.

[0162] In the embodiment shown, the autoencoder architecture 315 comprises, in addition to the encoder module 317 and the decoder module 319, a connection network 345 and a label network 343, each of which is designed as appropriately trained neural networks.

[0163] In the embodiment shown, for the training and / or for the execution of the already trained autoencoder architecture 315, the considered trajectory data 327, the map data 329, the position / trajectory data 353 of the ego vehicle 301 and the position / trajectory data 355 of the other road users are first converted into corresponding image data 339 of the trajectory data 327, image data 341 of the map data 329, image data 359 of the position / trajectory data 353 of the ego vehicle 301 and image data 361 of the position / trajectory data 355 of the other road users.

[0164] In the embodiment shown, when generating the image data 341 of the map data 329, the information of the data 331 or regarding the traffic volume, the prevailing traffic regulations, the road surface condition, the weather conditions, the time of day or the day of the week and the other Fig. The information described in paragraph 1 is taken into account.

[0165] The individual image data 339 of the trajectory data 327, as well as the image data 341 of the map data 329, comprise temporal sequences of image data 339, 341. The image data 339 of the trajectory data 327 describe temporal changes in the movement trajectories T1 of the respective observed road user 303 described by the trajectory data 327. The same applies to the image data 359 of the position / trajectory data 353 of the ego vehicle 301 and the image data 361 of the position / trajectory data 355 of the other road users, which each describe temporal changes in the positions of the ego vehicle 301 and the other road users, respectively. The individual image data 339, 359 can be provided by individual frames of a video recording.

[0166] In the embodiment shown, the image data 339 of the trajectory data 327 and the image data 341 of the map data 329 and the image data 359 of the position / trajectory data 353 of the ego vehicle 301 and the image data 361 of the position / trajectory data 355 of the other road users are first combined into a common data set, which serves as the input data set for the connection network 345. The connection network 345 is configured to generate corresponding input data 321 for the encoder module 317 based on the combined image data 339, 341, 353, 355.

[0167] The input data 321 includes the information from the image data 339 of the trajectory data and the image data 341 of the map data 329, the image data 359 of the position / trajectory data 353 of the ego vehicle 301, and the image data 361 of the position / trajectory data 355 of the other road users, and are each represented in vector form. The vector representation can include a representation in matrix format, for example, a 2D matrix.

[0168] In the embodiment shown, the output data of the connection network 345 includes position information of the road user 303, the ego vehicle 301, and the other road users. The position information can be at least 2D position information. Furthermore, the output data can include time information that defines the times at which the road user 303, the ego vehicle 301, and the other road users were positioned in the respective positions. The output data can be in a multidimensional tensor format. To generate the input data 321, the output data of the connection network 345 can be transformed into a vector form.

[0169] In addition to the information of the image data 339 of the trajectory data and the image data 341 of the map data 329 and the image data 359 of the position / trajectory data 353 of the ego vehicle 301 and the image data 361 of the position / trajectory data 355 of the other road users, the input data 321 includes labeling information 322.

[0170] The labeling information 322 contains the selection criteria, for example the position information in the form of the absolute and / or relative position information.

[0171] The labeling information 322 may also contain the classification information 357 in which the respective road users are assigned to corresponding classes of road users.

[0172] During the training process, the encoder module 317 is trained to generate corresponding latent space representations 326 of the trajectory data 327 based on the input data 321 and the labeling information 322. The latent space representations 326 take into account the information of the trajectory data 327 and the labeling information 322 and, depending on the embodiment, also the information of the map data 329, the position / trajectory data 353 of the ego vehicle 301, and the position / trajectory data 355 of the other road users.

[0173] The image data 341 of the map data 329, the image data 359 of the position / trajectory data 353 of the ego vehicle 301 and the image data 361 of the position / trajectory data 355 of the other road users also serve as input data for the label network 343. This is configured to generate the labeling information 322 based on the image data 341 of the map data 329, the image data 359 of the position / trajectory data 353 of the ego vehicle 301 and the image data 361 of the position / trajectory data 355 of the other road users.

[0174] The labeling information 322 serves as an addition to the sampled latent space representations 351. The sampled latent space representations 351 are sampled from the distributions of the latent space representations 326 generated by the encoder module 317 and serve as input data of the decoder module 319.

[0175] The labeling information 322 of the map data 329 enables easier assignment of the trajectory data 327 and, in particular, the associated latent space representations 326 to the appropriate classification information 357, position / trajectory data 353 of the ego vehicle 301, and the position / trajectory data 355 of the other road users. In particular, by labeling with the classification information, it can be achieved that when determining the distance between the latent space representations 334, 335, only movement trajectories belonging to road users of the same class are compared.

[0176] If road user 303 is identified as a motor vehicle, the respective latent space representations 334 of the motion trajectories T1 of road user 303 are only compared with latent space representations 335 of normal trajectories T2, which also originate from motor vehicles. The same applies if road user 303 is classified as a truck, bus, tram, motorcyclist, cyclist, or pedestrian.

[0177] Based on the latent space representations 326, 351, the decoder module 319 is trained to generate constructed input data 323, 234.

[0178] The connection network 345 as well as the label network 343 can each be designed as appropriately trained neural networks, in particular as convolutional networks.

[0179] Fig. 5 shows a flowchart of a method 100 for classifying a traffic behavior of a road user in the environment of an ego vehicle 301.

[0180] According to the invention, in a first method step 101, trajectory data 327 of a movement trajectory T1 is first received. The trajectory data 327 comprise at least position information of the respective movement trajectory T1. The trajectory data 327 can be based on environmental sensor data from the at least one environmental sensor 313 of the ego vehicle 301. The environmental sensor data can comprise, for example, camera data, LiDAR data, radar data, or acoustic data. The classification information 357 can be based on environmental detection based on the environmental sensor data.

[0181] In a further method step 103, at least one latent space representation 335 of at least one normal trajectory T2 is selected from the plurality of latent space representations 335 of a plurality of normal trajectories T2 in the latent space 333 based on the selection criterion, wherein the normal trajectories T2 are classified as movement trajectories based on normal traffic behavior.

[0182] The selection criterion can include absolute and / or relative position information. The position information indicates the location where the respective trajectory data of the motion trajectory T1 and / or normal trajectories T2 were recorded.

[0183] To select the at least one latent space trajectory 335, a search within the latent space 333 is carried out in a further method step 113 based on the absolute position information and / or based on the relative position information and the position information of the map display.

[0184] In a further method step 115, at least one latent space representation 335 or a predefined number of latent space representations 335 is selected, each of which has position information that is equal to the absolute and / or relative position information and / or the position information of the map representation of the ego vehicle 301 and / or the further road user 303 or has a deviation from the position information of the ego vehicle 301 and / or the further road user 303 that is less than or equal to a predetermined threshold value.

[0185] In a further method step 105, a distance D between the latent space representation 334 of the movement trajectory T1 of the road user 303 and the selected latent space representation 335 of the at least one normal trajectory T2 in the latent space 333 is determined. The normal trajectory T2 describes normal traffic behavior. The selected latent space representation 335 has the same value of the selection criterion as the latent space representation 334 of the movement trajectory T1, or the value differs from the value of the selection criterion of the latent space representation 334 of the movement trajectory by no more than a predetermined limit value.

[0186] In a further method step 107, the traffic behavior of the road user 303 is classified as normal traffic behavior if the distance D of the latent space representation 334 of the movement trajectory T1 of the road user 303 to the latent space representation 335 of the at least one normal trajectory T2 in the latent space 333 falls below a predetermined limit value.

[0187] In a further method step 109, however, the traffic behavior of the road user 303 is classified as abnormal traffic behavior if the distance D of the latent space representation 334 of the movement trajectory T1 of the road user 303 to the latent space representation 335 of the at least one normal trajectory T2 in the latent space 333 exceeds the predetermined limit value.

[0188] In a further method step 111, classification information is output in which the respective classification of the traffic behavior of the road user 303 is described.

[0189] Fig. 6 shows a further flowchart of the method 100 for classifying a traffic behavior of a road user in the environment of an ego vehicle 301 according to a further embodiment.

[0190] The embodiment shown is based on the embodiment in Fig. 5 and includes all process steps described there.

[0191] In the embodiment shown, in a further method step 117, map data 329 of a map display depicting the surroundings 305 of the ego vehicle 301 and / or position / trajectory data 353 of the ego vehicle 301 and / or position / trajectory data 355 of other road users are received.

[0192] In a further method step 119, classification information 357 is received. The classification information 357 defines classes of road users to which the respectively detected road user 303 is to be assigned. The classification information 357 assigns the road user 303 to a predefined class of road users. The road user 303 is thus identified as a motor vehicle, truck, bus, tram, motorcyclist, cyclist, pedestrian, or as another road user encountered in road traffic.

[0193] In a further method step 121, the trajectory data 327 are converted into image data 339 of the trajectory data 327 and / or the map data 329 are converted into image data 341 of the map data 329 and / or the position / trajectory data 353 of the ego vehicle 301 are converted into image data 359 of the position / trajectory data 353 of the ego vehicle 301 and / or the position / trajectory data 355 of other road users are converted into image data 361 of the position / trajectory data 355 of other road users.

[0194] Fig. 7 shows a flowchart of a method 200 for controlling an ego vehicle 301.

[0195] According to the invention, in a first method step 201, a classification of a traffic behavior of another road user located in an environment of the ego vehicle 301 is carried out by executing the method 100 according to the invention for classifying a traffic behavior of a road user in the environment of an ego vehicle 301 according to the embodiments described above.

[0196] In a further method step 203, the ego vehicle 301 is controlled according to the corresponding planned trajectory.

[0197] For this purpose, a safety maneuver is carried out in a further method step 205 if the traffic behavior of the other road user is classified as abnormal traffic behavior.

[0198] The safety maneuver may include reducing the speed of the ego vehicle 301 and / or increasing the distance between the ego vehicle 301 and the other road user and / or changing a lane of the roadway occupied by the ego vehicle. Alternatively or additionally, further maneuvers may be performed to increase the safety of the ego vehicle 301 and, in particular, to avoid a collision with the other road user.

[0199] The control of the ego vehicle 301 includes in particular the output of corresponding control signals from the computing unit to actuators of the ego vehicle 301 for executing the safety maneuver.

[0200] Fig.8 shows a schematic representation of a computer program product 400, comprising instructions which, when the program is executed by a computing unit, cause the computing unit to execute the method 100 for classifying a traffic behavior of a road user in the environment of an ego vehicle 301 and / or the method 200 for controlling an ego vehicle 301.

[0201] In the embodiment shown, the computer program product 400 is stored on a storage medium 401. The storage medium 401 can be any storage medium known from the prior art.

Claims

[1] Computer-implemented method (100) for classifying a traffic behavior of a road user (303) in an environment (305) of an ego vehicle (301), comprising: - receiving (101) trajectory data (327) of a movement trajectory (T1) of a road user (303) arranged in the environment (305) of an ego vehicle (301), wherein the trajectory data (327) comprise at least position information of the movement trajectory (T1) followed by the road user (303); - selecting (103) at least one latent space representation (335) of at least one normal trajectory (T2) from a plurality of latent space representations (335) of a plurality of normal trajectories (T2) in the latent space (333) based on a selection criterion, wherein the normal trajectories (T2) are classified as movement trajectories based on normal traffic behavior; - determining (105) a distance (D) of the latent space representation (334) of the movement trajectory (T1) of the road user (303) to the at least one latent space representation (335) of at least one normal trajectory (T2) in the latent space (333) selected according to the selection criterion; and - Classifying (107) a traffic behavior of the road user (303) as normal traffic behavior if the distance (D) of the latent space representation (334) of the movement trajectory (T1) of the road user (303) to the latent space representation (335) of the at least one normal trajectory (T2) in the latent space (333) falls below a predetermined limit value; or - Classifying (109) the traffic behavior of the road user (303) as an abnormal traffic behavior if the distance (D) of the latent space representation (334) of the movement trajectory (T1) of the road user (303) to the latent space representation (335) of the at least one normal trajectory (T2) in the latent space (333) exceeds the predetermined limit value, and - Outputting (111) classification information. [2] Method (100) according to claim 1, wherein the selection criterion comprises at least one item of position information of the ego vehicle (301) and / or the further road user (303). [3] Method (100) according to claim 2, wherein the position information of the ego vehicle (301) and / or the further road user (303) comprises at least one absolute position information in the form of GPS position coordinates. [4] Method (100) according to claim 2 or 3, wherein the position information of the ego vehicle (301) and / or the further road user (303) comprises at least one relative position information between the ego vehicle (301) and the further road user (303) and at least one position information of a map display depicting the surroundings (305) of the ego vehicle (301). [5] The method (100) of any one of the preceding claims 2 to 4, wherein selecting the latent space representation (335) comprises: Performing (113) a search within the latent space (333) based on the absolute position information and / or based on the relative position information and the position information of the map display; and Determining (115) at least one latent space representation (334), each of which has position information that is equal to the absolute and / or relative position information and / or the position information of the map representation of the ego vehicle (301) and / or the further road user (303) or has a deviation from the position information of the ego vehicle (301) and / or the further road user (303) that is less than or equal to a predetermined threshold value. [6] Method (100) according to one of the preceding claims 2 to 4, wherein the position information of the ego vehicle (301) and / or the further road user (303) is determined based on a starting point and / or an end point of the movement trajectory of the ego vehicle and / or the further road user, and wherein the starting point and / or the end point are determined with respect to a recording period of trajectory data of the ego vehicle (301) and / or the trajectory data of the further road user (303). [7] Method (100) according to one of the preceding claims, wherein the predetermined limit value is designed as a variable limit value, and wherein the limit value is designed such that a predetermined percentage of the latent space representations (334) of a latent space distribution (337) have a distance (D1) to a center of the latent space distribution (337) that is less than or equal to the predetermined limit value. [8] Method (100) according to one of the preceding claims, wherein the classification is carried out by a classification module (309), wherein the classification module (309) for determining the latent space representation (334) of the movement trajectory (T2) of the road user (303) comprises a correspondingly trained encoder module (317) of an autoencoder architecture (315), and wherein the autoencoder architecture (315) is trained to determine latent space representations (325, 326) of movement trajectories in the latent space (333) and to recognize and classify them as movement trajectories based on normal traffic behavior. [9] Method (200) for controlling an ego vehicle (301), comprising: Carrying out (201) a classification of a traffic behavior of a road user (303) in the environment of an ego vehicle (301) based on trajectory data (327) of a movement trajectory (T1) of the road user (303) according to the method (100) for classifying a traffic behavior of a road user (303) in the environment (305) of an ego vehicle (301) according to one of the preceding claims 1 to 8; and Controlling (203) the ego vehicle (301) by executing at least one control function of the ego vehicle (301) taking into account the classification of the traffic behaviour of the road user (303). [10] The method (200) of claim 9, wherein controlling the ego vehicle (301) comprises: Carrying out (205) a safety maneuver if the traffic behavior of the road user (303) is classified as abnormal traffic behavior. [11] Computing unit (307) which is configured to carry out the method (100) for classifying a traffic behavior of a road user (303) in the environment (305) of an ego vehicle (301) according to one of the preceding claims 1 to 8 and / or the method (200) for controlling an ego vehicle (301) according to one of the preceding claims 9 to 10. [12] Computer program product (400) comprising instructions which, when the program is executed by a data processing unit, cause the data processing unit to execute the method (100) for classifying a traffic behavior of a road user (303) in the environment (305) of an ego vehicle (301) according to one of the preceding claims 1 to 8 and / or the method (200) for controlling an ego vehicle (301) according to one of the preceding claims 9 to 10.

Images