Method for generating knowledge graph for traffic motion prediction, method for traffic motion prediction, and method for controlling host vehicle

Abstract

The method relates to a computer-implemented method for generating a knowledge graph for traffic motion prediction, the method comprising: receiving environmental sensor data of at least one environmental sensor of a host vehicle; receiving map data from the electronic road map; extracting information about the at least one traffic participant from the environmental sensor data, and extracting information about a movement trajectory where the traffic participant is located from the map data; and generating a knowledge map of the road network in an environment of the host vehicle based on the map data and / or the environment sensor data, the knowledge map comprising nodes and edges, the knowledge graph includes at least one node representing the traffic participant and at least one node representing a lane in which the traffic participant is located. The invention also relates to a method for traffic motion prediction. The invention also relates to a method for controlling a host vehicle.

Description

Technical Field

[0001] The present invention relates to a method for generating a knowledge graph for traffic movement prediction. The present invention also relates to a method for traffic movement prediction. The present invention also relates to a method for controlling a vehicle. Background Art

[0002] Motion prediction in traffic scenarios involves accurately predicting the behavior of surrounding traffic participants or other moving objects. To intuitively achieve this task, multiple sources of information need to be considered, including the vehicle's driving path, road topology, lane dividers, crosswalks, and traffic regulations. Although previous studies have demonstrated the potential of leveraging heterogeneous context to improve motion prediction, state-of-the-art deep learning methods still rely on a limited subset of this information. This is mainly due to the limited availability of comprehensive representations of contextual information available in prominent datasets.

[0003] Traffic motion prediction is an essential component of autonomous driving, as it enables autonomous vehicles to anticipate the movements of other traffic participants and avoid dangerous situations that could lead to collisions. Machine learning, and in particular deep learning methods, have proven to be very successful in developing motion prediction and can generally achieve high accuracy. Easily accessible datasets are essential for progress in machine learning. MNIST, COCO, and ImageNet have been crucial for advances in computer vision, GLUE and SQuAD for natural language understanding, and the training environments MuJoCo and OpenAI Gym for reinforcement learning, to name a few. However, in recent years, the limitations of deep learning models have been the subject of much research.

[0004] Specifically, deep learning models lack robustness, interpretability, and generalization to new domains. One assumption underlying the lack of robustness of modern deep learning models is that they are purely sub-symbolic and statistical, learning correlations between input features and target variables rather than gaining a causal, structured understanding of the task.

[0005] Humans do not reason at the level of individual pixels when driving, but at the level of objects, perceived intentions, and spatial and semantic relationships. Humans have inherent prior knowledge, for example, understanding that no traffic participant can have an acceleration above a certain threshold. When deep learning models are trained end-to-end scenarios from raw data, high-level structured information (knowledge) is usually lost. In addition, for practical applications of motion prediction algorithms, it is crucial to consider safety aspects. The international standard ISO26262 for functional safety of road vehicles needs to be met. It requires that the behavior of components is fully specified and verified. Summary of the invention

[0006] The object of the present invention is to provide an improved method for generating a knowledge graph for traffic movement prediction and an improved method for traffic movement prediction and an improved method for controlling a vehicle.

[0007] The object is achieved by the method of the independent claim. Further embodiments are subject matter of the dependent claims.

[0008] According to one aspect of the present invention, there is provided a computer-implemented method for generating a knowledge graph for traffic movement prediction, the method comprising:

[0009] receiving environmental sensor data of at least one environmental sensor of a host vehicle, wherein the environmental sensor data represents an environment of the host vehicle and comprises information about at least one traffic participant located in the environment of the host vehicle;

[0010] receiving map data from an electronic road map, wherein the map data represents a road network in the environment of the host vehicle and includes information about at least one movement trajectory of the traffic participant;

[0011] extracting information about the at least one traffic participant from the environmental sensor data, and extracting information about the road and / or lane on which the traffic participant is located from the map data; and

[0012] A knowledge graph of the road network in the environment of the vehicle is generated based on the map data and / or the environmental sensor data, the knowledge graph comprising nodes and edges, wherein the knowledge graph comprises at least one node representing the traffic participant and at least one node representing the lane in which the traffic participant is located.

[0013] By this, a technical advantage can be achieved, namely an improved method for generating a knowledge graph for traffic movement prediction can be provided. To this end, environmental sensor data of at least one environmental sensor of the host vehicle are considered as information of the knowledge graph. The environmental sensor data represent the environment of the host vehicle and include information about at least one traffic participant located in the environment of the host vehicle. Via the environmental sensor data, information about the at least one traffic participant located in the environment of the host vehicle can be extracted.

[0014] Furthermore, map data from an electronic road map are received, which represents a road network in the environment of the host vehicle. Via the map data, information about at least one movement trajectory of a traffic participant located in the environment of the host vehicle can be extracted.

[0015] Via environmental sensor data, information about individual characteristics of traffic participants can be taken into account for generating the knowledge graph. Via map data of an electronic road map, information about characteristics of the movement trajectories of traffic participants can be taken into account when generating the knowledge graph.

[0016] During generation of the knowledge graph, information extracted from the environmental sensor data and information extracted from the map data are included in the knowledge graph by means of nodes and / or edges of the knowledge graph.

[0017] Thus, a knowledge graph can be generated that includes information about individual characteristics of traffic participants located in the environment of the host vehicle and information about individual characteristics of the movement trajectories of those traffic participants. This allows the generation of a knowledge graph with precise information about traffic participants located in the environment of the host vehicle and the corresponding movement trajectories of these traffic participants.

[0018] Based on the knowledge graph, accurate and reliable traffic movement predictions can be made for the future movements of traffic participants in the environment of the host vehicle.

[0019] In an applied sense, a movement trajectory of a traffic participant is defined as a distinguishable area in a road network. A movement trajectory is a part of the infrastructure in the environment of the host vehicle, on which the respective traffic participant is located and which enables the traffic participant to carry out a future movement.

[0020] According to one embodiment, the movement trajectory of the traffic participant is at least one of the following list: roads and / or lanes, intersections, underpasses, bridges, highways, highway entrances, highway exits, roundabouts, parking lots, parking lots, bicycle lanes, tram tracks, crosswalks, and sidewalks.

[0021] Thereby, the technical advantage can be achieved that a knowledge graph can be generated taking into account a wide variety of different movement trajectories for different traffic participants located in the environment of the host vehicle. In this way, a knowledge graph can be generated for a wide variety of different traffic participants located at different locations within the environment of the host vehicle. This allows for an accurate and flexible prediction of traffic movements based on a corresponding knowledge graph that is applicable to many different traffic scenarios.

[0022] According to one embodiment, the map data of the electronic road map comprise further information about further features of the movement trajectory of the traffic participant, and wherein at least one further feature of the movement trajectory of the traffic participant is integrated into the knowledge graph via at least one further node.

[0023] This provides the technical advantage that, via further features of the movement trajectories of traffic participants, additional information about the movement trajectories can be introduced into the knowledge graph, which allows for a more accurate prediction of traffic movements based on the corresponding knowledge graph.

[0024] According to one embodiment, the environmental sensor data also comprise further information about at least one further feature of the traffic participant, and wherein the at least one further feature of the traffic participant is integrated into the knowledge graph via at least one further node.

[0025] As a result, the technical advantage can be achieved that further information about additional individual characteristics of the traffic participants can be introduced into the knowledge graph. Due to this further information, a more precise description of the traffic participants can be introduced into the knowledge graph. Therefore, a more precise movement prediction of the future movement of the corresponding traffic participants can be achieved based on the corresponding knowledge graph.

[0026] According to one embodiment, nodes of the knowledge graph and further nodes are organized into classes and subclasses.

[0027] Thus, a technical advantage may be achieved in that information within a knowledge graph is organized by organizing the nodes of the knowledge graph into classes and subclasses.

[0028] According to one embodiment, the motion trajectory is further characterized by at least one of the following list, the list comprising: road geometry, lane geometry, lane dividers, lane boundaries, lane connectors, intersections, stop areas, traffic signals, traffic signs, traffic rules, road conditions, slope values, crosswalks, parking areas, road sections, roadblocks.

[0029] This allows the technical advantage that a knowledge graph can be generated for a plurality of different traffic participants in a plurality of different movement trajectories in the environment of the host vehicle, which allows a flexible and possibly applicable traffic movement prediction based on the corresponding knowledge graph.

[0030] According to one embodiment, a further characteristic of the traffic participant is at least one of the following list, which list includes: static object, moving object, person, animal, vehicle, car, truck, tram, motorcycle, bicycle, obstacle, traffic cone, relative position to at least one other traffic participant and / or to the vehicle.

[0031] Thereby, the technical advantage can be achieved that a variety of different types of traffic participants can be considered to generate corresponding knowledge graphs. This allows for a widely applicable traffic movement prediction based on the corresponding knowledge graph.

[0032] According to one embodiment, the method further comprises:

[0033] Determine timestamps for environmental sensor data and map data;

[0034] organizing the environmental sensor data and the map data into scenes and sequences, wherein a scene includes information of the environmental sensor data and the map data for a timestamp, and wherein a sequence is a series of consecutive scenes; and

[0035] The nodes of the knowledge graph are organized with respect to scenes and sequences of the environmental sensor data and the map data.

[0036] Thus, a technical advantage can be achieved, namely that since the nodes of the knowledge graph are organized relative to the scenes and sequences of the environmental sensor data and the map data, time instances can be introduced into the knowledge graph. This allows the use of the knowledge graph for traffic movement prediction based on time-related data, because the time-related data appears on the changing traffic scenes that exist during the operation of the host vehicle. Therefore, the corresponding knowledge graph can be used for traffic movement prediction, and the corresponding knowledge graph can be applied during the operation of the host vehicle.

[0037] According to one embodiment, the information of the map data taken into account when generating the knowledge graph is limited to the area of ​​possible paths of traffic participants.

[0038] Thereby, a technical advantage can be achieved that the size of the knowledge graph can be reduced and reduced to information essential for traffic movement prediction. This allows for fast and efficient traffic movement prediction.

[0039] According to one embodiment, the method is performed during a driving operation of the host vehicle.

[0040] Thereby, the technical advantage can be achieved that a knowledge graph can be generated and updated during operation of the host vehicle. This allows accurate traffic movement predictions.

[0041] According to one aspect, there is provided a computer-implemented method for traffic movement prediction, the method comprising:

[0042] generating a knowledge graph for traffic movement prediction by executing the method for generating a knowledge graph for traffic movement prediction according to any of the aforementioned embodiments; and

[0043] A future movement of at least one traffic participant in the environment of the host vehicle is predicted based on the knowledge graph by a movement prediction module.

[0044] Thereby, a technical advantage can be achieved, namely an improved method for traffic movement prediction can be provided.The method for controlling a host vehicle uses the method for generating a knowledge graph for traffic movement prediction having the above technical advantages.

[0045] According to one embodiment, the movement prediction module includes a trained artificial intelligence capable of predicting the movement of traffic participants based on the information of the knowledge graph.

[0046] Thereby, a technical advantage can be achieved, namely a robust and efficient prediction module can be provided.

[0047] According to one embodiment, the artificial intelligence includes a graph neural network.

[0048] Thereby, a technical advantage can be achieved, namely a robust and efficient prediction module can be provided.

[0049] According to one aspect, there is provided a computer-implemented method for controlling a host vehicle, the method comprising:

[0050] Predicting the future movement of at least one traffic participant in the environment of the host vehicle by executing the method for traffic movement prediction according to any one of the aforementioned embodiments; and

[0051] At least one control function of the host vehicle is performed based on the predicted movement of the at least one road user.

[0052] Thereby, a technical advantage can be achieved, namely an improved method for controlling a host vehicle can be provided.The method for controlling a host vehicle uses the method for traffic movement prediction having the above-mentioned technical advantages.

[0053] According to one aspect, a computing unit is provided, which is equipped to execute a method for generating a knowledge graph for traffic movement prediction according to any of the aforementioned embodiments and / or a method for traffic movement prediction according to any of the aforementioned embodiments and / or a method for controlling a vehicle.

[0054] According to one aspect, a computer program product is provided, comprising instructions which, when the program is executed by a data processing unit, cause the data processing unit to execute a method for generating a knowledge graph for traffic movement prediction according to any one of the aforementioned embodiments and / or a method for traffic movement prediction according to any one of the aforementioned embodiments and / or a method for controlling a vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Embodiments of the present invention will be described with respect to the following drawings. The drawings show:

[0056] Figure 1 is a schematic diagram of a system for generating a knowledge graph for traffic movement prediction according to one embodiment;

[0057] Figure 2 is another schematic diagram of a system for generating a knowledge graph for traffic movement prediction according to another embodiment;

[0058] Figure 3 is another schematic diagram of a system for generating a knowledge graph for traffic movement prediction according to another embodiment;

[0059] Figure 4 is a schematic diagram of a knowledge graph for traffic movement prediction according to one embodiment;

[0060] Figure 5 is a flow chart of a method for generating a knowledge graph for traffic movement prediction according to one embodiment;

[0061] Figure 6 is a flow chart of a method for traffic movement prediction according to one embodiment;

[0062] Figure 7 is a flow chart of a method for controlling a host vehicle according to one embodiment; and

[0063] Figure 8 It is a schematic diagram of a computer program product. DETAILED DESCRIPTION

[0064] Figure 1 A schematic diagram of a system 500 for generating a knowledge graph 400 for traffic movement prediction according to one embodiment is shown.

[0065] Figure 1 Two traffic scenes are shown, represented in the first scene 533 and the second scene 535 shown in the diagrams a) and b). The first scene 533 and the second scene 535 have time stamps t1, t2 at which the respective traffic scenes were observed.

[0066] In both traffic scenarios, the host vehicle 501 is shown traveling on a road 517 comprising a first lane 519 and a second lane 547. In both traffic scenarios shown, another traffic participant 507 is located in the environment of the host vehicle 501. In the shown embodiment, the other traffic participant 507 is a vehicle 565, in particular a car 531. The traffic participant 507 moves on a movement trajectory 515.

[0067] The movement trajectory 515 is defined by a section of infrastructure in the environment of the host vehicle, which allows for the future movement of the traffic participant 507. In the example shown, the movement trajectory 515 is given by a road 517, in particular by one of the lanes 519, 547.

[0068] In the two scenarios 533, 535, driving scenarios of the host vehicle 501 and another traffic participant 507 are shown. The two scenarios 533, 535 describe two consecutive traffic scenarios, wherein, from the scenario 533 to the scenario 535, the traffic participant 507 performs a lane change maneuver and changes its position from the first lane 519 to the second lane 547. The two traffic scenarios shown are only examples and should not limit the scope of the present invention.

[0069] In the illustrated embodiment, the host vehicle 501 comprises a computing unit 541 and at least one environmental sensor 505. Via the environmental sensor 505, the host vehicle 501 is able to collect environmental sensor data 503 representing the environment of the host vehicle 501. The environmental sensor 505 can be, for example, a camera sensor, an ultrasonic sensor, a radar sensor, or any environmental sensor known in the field of autonomous driving. Based on the environmental sensor data 503, the host vehicle 501 is able to detect another traffic participant 507 located in the environment of the host vehicle 501.

[0070] In addition, the host vehicle 501 can access an electronic road map 511. The electronic road map 511 represents a road network 513 having a plurality of roads, lanes, and other features of traffic infrastructure, and provides detailed information about the roads, lanes, and infrastructure in the environment of the host vehicle 501.

[0071] By using the computing unit 541 , the map data 509 and the environmental sensor data 503 , the host vehicle is able to perform the method according to the present invention for generating a knowledge graph for traffic movement prediction.

[0072] For this purpose, ego vehicle 501 detects road users 507 and corresponding movement trajectories 515 in the environment of ego vehicle 501 via environmental sensor data 503 .

[0073] Further information about detected and / or identified movement trajectory 515 of road participant 507 is received via map data 509 of electronic road map 511 .

[0074] Based on environmental sensor data 503 and map data 509 , movement trajectory 515 of traffic participant 507 can be identified, for example, as road 517 and / or lanes 519 , 547 , intersections, underpasses, bridges, motorway entrances, motorway exits, roundabouts, parking spaces, parking lots.

[0075] Based on the environmental sensor data, traffic participant 507 can also be identified as a static object, a moving object, a person, an animal, a vehicle 565 , a car 531 , a truck, a tram, a motorcycle, a bicycle, an obstacle or a traffic cone.

[0076] Depending on the type of traffic participant 507 , movement trajectory 515 may also be identified as a bicycle path, a tram track, a pedestrian crossing or a sidewalk or any other possible movement trajectory 515 .

[0077] Based on the map data 509, additional individual information about previously identified motion trajectories 515 may be taken into account. Further features may include road geometry, lane geometry, lane dividers 525, lane boundaries 527, lane connectors, intersections, stop areas, traffic signals, traffic signs, traffic regulations, road conditions, slope values, pedestrian crossings, parking areas, road sections, road barriers or any other feature of the infrastructure presenting the traffic scene.

[0078] In order to generate a knowledge graph based on information from the map data 509 and the environmental sensor data 503 , the above-mentioned information about the motion trajectory 515 and the participant trajectory 507 is extracted from the environmental sensor data 503 and the map data 509 .

[0079] Based on the extracted information about the movement trajectory 515 and the traffic participant 507 , a knowledge graph is generated, wherein the extracted information is included in the knowledge graph via a plurality of nodes and edges connecting the nodes.

[0080] According to one embodiment of the present invention, the corresponding knowledge graph is organized in the above-mentioned scenes 533, 535, each of which describes a subsequent traffic scene occurring at a different time stamp t1, t2. In this way, the knowledge graph is organized in a time-dependent representation, wherein for each scene 533, 535 and the corresponding time stamp t1, t2, the above-mentioned information is included in the knowledge graph with corresponding nodes and edges.

[0081] exist Figure 1 , two illustrated scenes 533, 535 define a sequence 537. The sequence 537 describes the sequential dependency of the illustrated scenes 533, 535. Sequence information can also be included in the knowledge graph as additional information.

[0082] In both figures a) and b), in addition to the road 517 and lanes 519 , 547 , lane dividers 525 and lane boundaries 527 are also shown.

[0083] In the illustrated embodiment, the host vehicle 501 further comprises a motion prediction module 539 installed on the computing unit 541. Via the motion prediction unit 531, the host vehicle 501 is able to perform the method for traffic motion prediction according to the present invention.

[0084] In order to perform traffic movement prediction, during operation, host vehicle 501 collects environmental sensor data 503 in order to detect and identify traffic participants 507 located in the environment of host vehicle 501 and corresponding movement trajectories 515 of traffic participants 507 .

[0085] Furthermore, information about a road network 513 within the environment of the host vehicle 501 is extracted from the map data 509 of the electronic road map 511. Based on the information extracted from the map data 509 and the environmental sensor data 503, the host vehicle 501 generates the above-mentioned knowledge graph. Such generation of the knowledge graph can be performed during an operating mode of the host vehicle 501, in particular a driving mode.

[0086] Based on the information of the knowledge graph, the traffic prediction module 539 can predict the future movement of the traffic participants 507 located and identified in the environment of the host vehicle 501. To this end, the traffic prediction module 539 can be designed as a trained artificial intelligence that can perform movement prediction based on the information of the above-mentioned knowledge graph. To this end, the movement prediction module 539 can be designed as a graph network that can perform the movement prediction.

[0087] Via the computing unit 541, the host vehicle 501 can also execute the method according to the invention for controlling the host vehicle 501. To this end, the host vehicle 501 first executes the method for traffic movement prediction and predicts the future movement of other road users 507 located and identified in the environment of the host vehicle 501. Based on the predicted future movement of the identified road users 507, the host vehicle 501 executes at least one control function. The control function can be, for example, an acceleration or deceleration and / or a steering function.

[0088] Figure 2 Another schematic diagram of a system 500 for generating a knowledge graph 400 for traffic movement prediction according to another embodiment is shown.

[0089] Figure 2 A different traffic scenario is shown as another example of the present invention. In the traffic scenario shown, another traffic participant 507 is located in the environment of the host vehicle 501. In the current scenario, the other traffic participant 507 is a person 529. The person 529 moves on his movement trajectory 515. In the current example, the movement trajectory 515 is a crosswalk 523 that intersects with the lane 519 of the road 517 on which the host vehicle 501 is located. In addition to the differences mentioned, Figure 2 The embodiment shown in Figure 1 and includes Figure 1 All features shown in .

[0090] Figure 3Another schematic diagram of a system 500 for generating a knowledge graph 400 for traffic movement prediction according to another embodiment is shown.

[0091] Figure 3 Graph a) shows another traffic scene. Graph b) shows a schematic diagram of the same traffic scene. In the traffic scene shown, host vehicle 501 is located in first lane 519. Traffic participant 507 is located in second lane 547. In addition, first additional traffic participant 543 and second additional traffic participant 545 are located in fourth lane 551 of roundabout 521.

[0092] In the traffic scenario shown, a first further road user 543 and a second further road user 545 are located on the same lane 551 and therefore have a longitudinal position 553 relative to one another.

[0093] Host vehicle 501 and road user 507 are located on different lanes oriented parallel to each other. Therefore, host vehicle 501 and road user 507 have a lateral relative position 555 to each other.

[0094] The second lane 547 where the road user 507 is located and the fourth lane 551 of the roundabout 521 where the first other road user 543 and the second other road user 545 are located intersect each other. Therefore, the road user 507 and the first other road user 543 and the second other road user 545 have an intersecting relative position 557.

[0095] In diagram b), the relative positions of host vehicle 501 , road user 507 , and first further road user 543 and second further road user 545 are shown in a schematic manner.

[0096] According to another exemplary embodiment of the present invention, further information about the traffic participant 507 or traffic participants 507 , 543 , 545 in the environment of the host vehicle 501 may include the above-mentioned relative positions relative to each other and relative to the host vehicle 501 .

[0097] Figure 4 A schematic diagram of a knowledge graph 400 for traffic movement prediction according to one embodiment is shown.

[0098] The knowledge graph 400 includes a plurality of nodes 401 and edges 403 connecting the nodes 401 .

[0099] exist Figure 4 On the right side of , information about road users 507 is shown with corresponding nodes 401. Figure 4 On the left side of , information about movement trajectory 515 of road participant 507 is considered with corresponding nodes 401 and edges 403 .

[0100] On the right, road user 507 is represented by node 401. A plurality of further nodes are connected to the node of road user 507. Via these further nodes, road user 507 can be characterized more precisely.

[0101] For example, traffic participant 507 can be characterized as static object 561 or movable object 683. Both objects 561, 683 are included in knowledge graph 400 via separate nodes 401. Static object 561 can be, for example, bicycle rack 563. Movable object 683 can be, for example, obstacle 583, pushable / pullable object 585, traffic cone 587, debris 581. Each object type is provided with a corresponding separate node 401.

[0102] In addition to static objects 561 or movable objects 683, traffic participants 507 may also be vehicles 565. Vehicle 565 may be bicycle 567, motorcycle 569, truck 571, or car 531. Each vehicle type is provided with a corresponding node 401 and is connected to the node of vehicle 565 via a corresponding edge.

[0103] In addition to vehicle 565, traffic participant 507 may also be animal 573 or person 529. Person 529 may be a police officer 575, an adult 577, or a child 579.

[0104] All nodes 401 on the right side of the knowledge graph 400 are features of the traffic participant 507 and are organized into subclasses 559. Therefore, each node 401 is connected to each node 401 of the main class and / or to the traffic participant 507 via a subclass relationship 559.

[0105] The knowledge graph 400 is organized in a first scenario 533. All nodes 401 shown are organized with a corresponding scenario 533. For multiple scenarios 533, 535, the knowledge graph 400 may include Figure 4 Multiple layers of nodes 401 are shown. Each layer describes a separate scene 533, 535 and is connected to layers representing previous and / or subsequent scenes.

[0106] This connection of scene 533 to previous and / or subsequent scenes is shown in knowledge graph 400 via hasPreviousScene relationship 599 , and the connection to the subsequent scene is shown via hasNextScene relationship 597 .

[0107] The previous scene and the next scene are associated with respective time stamps t1 , t2 at which respective map data 509 and environment sensor data 503 have been connected.

[0108] The scene 533 is part of the sequence 537 and is connected to the corresponding node 401 of the sequence 537 via a hasFirstScene relationship 601 , a hasLastScene relationship 603 , and a hasScene relationship 605 .

[0109] Sequence 537 represents multiple subsequent scenes 533. Sequence 537 is a part of trip 607. Trip 607 includes multiple sequential sequences 537 and is connected to the nodes of sequence 537 via hasSequence relationship 611.

[0110] The trip 607 is connected to the node 401 of the ego vehicle 501 via the hasEgoVehicle relationship 609 .

[0111] The node 401 of the scene 533 is also connected to the node of the scene participant 589 via a hasSceneParticipant relationship 595. The scene participant 589 is also connected to the traffic participant 507 via an isSceneParticipant relationship 591.

[0112] A scene participant describes a traffic participant 507 of a specific scene 533, 535. The number of traffic participants 507 is located with respect to the corresponding traffic scene.

[0113] The scene participant 589 is also connected to itself in the nextScene via the nextScene relationship 593. The scene participant 589 is also connected to the point 613 via the hasPosition relationship 685. This may describe the specific position of the scene participant 589.

[0114] Point 613 is connected to node 401 of geometry 635 via a subclass relationship 559. Point 613 is also connected to lane 519 via an isPointOnLane relationship 657. Lane 519 is connected to node 401 of motion track 515 via a subclass relationship 559.

[0115] Lane 519 is also connected to the next lane via hasNextLane relationship 617. Lane 519 may also be the right lane or the left lane, thus including hasLeftLane relationship 687 and hasRightLane relationship 615.

[0116] Lane 519 may also be a subclass of road-crossing element 619 , and therefore connected to corresponding node 401 of road-crossing element 619 via subclass relationship 559 .

[0117] Lane 519 is also connected to roadblock 627 via IsLaneOnRoadBlock relationship 645 . Roadblock 627 is connected to road element 653 via subclass relationship 559 .

[0118] Roadblock 627 is also connected to road segment 623 via isRoadBlockOnRoadSegment relationship 629 .

[0119] The road segment 623 is also connected to the lane connector 621 via the isConnectorOnRoadSegment relationship 625. The road segment 623 is also connected to the intersection 681 via the subclass relationship 559.

[0120] Lane connector 621 is also connected to lane 519 via ConnectsIncomingLane relationship 647 and ConnectsOutgoingLane relationship 649 .

[0121] Lane 519 is also connected to parking area 631 via CarparkAreaIsNextTo relationship 633 .

[0122] The parking area 631 is also connected to the lane divider via the carparkAreahasShape relationship 679 .

[0123] Lane 519 is also connected to polygon 637 via LaneHasShape relationship 639 .

[0124] Lane 519 is also connected to lane divider 525 via hasRightLaneDivider relationship 641 and hasLeftLaneDivider relationship 643 .

[0125] The road element 653 also has an isPartOf relationship 651 . Furthermore, the road element 653 is connected to the lane segment 655 via a subclass relationship 559 .

[0126] Lane segment 655 also has a SwitchViaSingleSolidYellow relationship 659 , a SwitchViaRoadDivider relationship 661 , a SwitchViaDoubleSolidYellow relationship 663 , a SwitchVia relationship 665 , a SwitchViaSingleZigZagWhite relationship 667 , a SwitchViaSingleWhite relationship 669 , a SwitchViaNonVisible relationship 671 , a SwitchViaDoubleDashedWhite relationship 673 , and is connected to the next lane segment via a hasNextLaneSnipped relationship 675 .

[0127] Lane snippet 655 is also connected to lane 519 via hasLaneSnipped relationship 677 .

[0128] Figure 4 The embodiment shown is only an example of a possible knowledge graph 400 for traffic movement prediction. According to the present invention, the knowledge graph 400 includes at least information about a traffic participant 507 in the environment of the host vehicle 501 and information about the movement trajectory 515 of the specific traffic participant 507. The corresponding information is included in the knowledge graph 400 via a plurality of nodes 401 and edges 403.

[0129] As described above, the structure of the knowledge graph 400, in particular the information stored in the nodes 401 and the corresponding relationships between the nodes 401, depends on the corresponding traffic scenario, the number of traffic participants 507 located in the environment of the vehicle 501, and the information accessible via the environmental sensor data 503 and the map data 509.

[0130] In the present invention, the concepts of sequence 537 and scene 533, 535 are used. A detailed description of the concepts of sequence 537 and scene 533, 535 is provided in Knowledge graphs for automated driving by Lavdim Halilaj, Jürgen Luettin, Cory Andrew Henson, Sebastian Monka, 2022 IEEE Fifth International Conference on Artificial and Knowledge Engineering (AIKE), pages 98-105, 2022.

[0131] In short, scenes 533, 535 refer to a single moment in a traffic situation. The relation hasTimestamp is an integer data attribute with a unix timestamp defining a specific moment.

[0132] Sequence 537 is an ordered collection of scenes 533, 535. Sequence 537 can be considered as a video, the frames of which are scenes 533, 535. Since the order of scenes 533, 535 is inherent to them, object properties hasNextScene 597 and hasPreviousScene 599 are defined to link consecutive scenes 533, 535. Sequence 537 and scene 533, 535 instances can be generated from map data 509 of electronic road map 511. Map 511 can be based on nuScenes dataset.

[0133] Equations (1) to (22) are expressed in the form of description logic. A detailed explanation of the form can be found in Found in etal.arXiv:1201.4089.

[0134] Equation (1) provides the definition of scenes 533 , 535 and sequence 537 . Scenes 533 , 535 and sequence 537 are all nodes 401 in knowledge graph 400 .

[0135]

[0136] Sequence 537 refers to a specific motion prediction situation. During the recording of motion data, the vehicle may travel for several hours and record several sequences 537.

[0137] A trip 607 is a recording session, each of whose entities refers to a number of sequences 537. Each trip 607 is made in a specific area of ​​interest, a location associated with it via hasLocation. hasRightHandTraffic is a Boolean property describing the driving direction of the location.

[0138] Equations (3) and (4) provide definitions of the conceptual itinerary 607 and location (Location).

[0139]

[0140] Trip 607 instances may be generated from map data 509 (eg, nuScenes log records), and location instances may be manually created for each of the four maps.

[0141] The traffic participant 507 concept represents a traffic agent that exists in one or more scenes 533, 535. Various types of participants are modeled as subclasses of the participant concept. There are 23 different types of participants in total.

[0142] For example, Car, Adult, Child, Policeman, Ambulance, Bicycle, etc. An actor can refer to an entity at a certain time step. A new relation inNextScene is introduced to be able to link entities across time.

[0143] Furthermore, the concept SceneParticipant is introduced as a concept of an agent at a specific time step, and the meaning of the traffic participant 507 is changed to generally represent an agent independent of time.

[0144] This avoids having to redundantly store time-irrelevant information, such as the size of the agent. Semantic relationships between scene participants can be modeled so that agents can follow each other, possibly intersect or run parallel to each other.

[0145] Equations (5) and (6) provide the definitions of the concepts SceneParticipant and Participant.

[0146]

[0147] The SceneParticipant instance is generated by the SAMPLE_ANNOTATION and EGO_POSE records. The EGO_POSE record is needed so that the host vehicle can be included as a scene participant. This is novel in the data representation provided. Previous work has ignored the influence of the host vehicle 501 on the motion of the target vehicle.

[0148] Data analysis of the currently provided nuScenes dataset shows that in a large number of cases, the host vehicle 501 and the target can be as close as 2 meters. Therefore, the host vehicle 501 is considered to have a high influence on the behavior of the target vehicle.

[0149] A central component of road traffic infrastructure is the lane 519, 547, 549, 551. This is defined as a non-overlapping stretch of the road surface, usually bounded by lane boundaries, allowing only one direction of driving. This is the natural, physical formalization of lanes, as people do not consider intersections or merge points to be part of a specific lane 519, 547, 549, 551.

[0150] A different modeling option would be to use a definition that treats lanes 519, 547, 549, 551 as logical entities that can overlap rather than physical entities. Such lanes 519, 547, 549, 551 would cross the intersection. We chose a natural physical definition to be compatible with nuScenes. To keep the logical connection information with a natural definition, LaneConnectors are needed with the functional properties hasIncomingLane and hasOutgoingLane pointing to Lane respectively.

[0151] Equations (7) and (8) provide the definitions of the concepts Lane and Lane Connector.

[0152]

[0153] Lane and LaneConnector instances are generated from LANE and LANECONNECTOR records accordingly. LaneSnippet, switchVia. Lane boundaries are another key factor in determining how a car should drive.

[0154] There are different lane separator types, such as solid and dashed. LaneSnippet is defined as a portion of lane 519, 547, 549, 551, with a single border type on each of its left and right sides. This allows the introduction of switchVia attributes for each type of border, i.e., switchViaDoubleDashed, switchViaSingleSolid, etc. For example, there is a single solid border between adjacent segments, and the adjacent segments are related to each other via switchViaSingleSolid, indicating that a single solid border must be crossed to switch from one of the adjacent segments to another.

[0155] Illegal switches via boundaries are retained in the model because cars may sometimes violate traffic rules, such as overtaking across a physical boundary. hasNextLaneSnippet points from one snippet to the next snippet immediately below, keeping them in order, and hasLaneSnippet connects the snippets to their parent lanes 519, 547, 549, 551.

[0156] In addition, since experimental evidence shows that keeping segments short is important for trajectory prediction performance, a segment is further divided if its length exceeds 20 meters. snippetHasLength records the length of a particular lane segment.

[0157] Equation (9) provides the definition of the concept LaneSnippet.

[0158]

[0159] LaneSnippet instances are computed from the LANE records. The boundary type (solid, dashed, etc.) on each side of the lane is tracked and segmented into segments with constant boundary type on either side. Segments are split if necessary to meet the 20m length constraint. This produces LaneSnippet instances with constant boundary type on both sides. switchVia edges are placed between adjacent LaneSnippet instances.

[0160] To represent the center lines of lanes and lane connectors (where cars usually drive), a series of poses are used. A pose consists of a position and an orientation. The orientation here represents the direction of the lane at a certain location, i.e. the direction of traffic. OrderedPose is a subclass of Pose, which also has a textithasNextPose property.

[0161] This is used to order them, defining a typical trajectory along the LaneConnector via the connectorHasPose relations to all ordered poses. The position of the Pose is modeled using sf:Point like the agent position, and its orientation is modeled with the data attribute poseHasOrientation expressed as the angle between the positive x-axis and the facing (yaw) direction. In contrast to lane connectors, lane models also need to satisfy the capacity issue regarding width.

[0162] The natural naming of LaneSlice was chosen to represent the combination of center pose and lane width. hasNextLaneSlice keeps the order of consecutive slices by concatenating them, hasLaneSlice points to its slice from the parent lane, and laneSliceHasWidth is the data property that the name implies.

[0163] Equations (10), (11), and (12) provide the definitions of the concepts LaneSlice and OrderedPose.

[0164]

[0165] OrderedPose instances are generated from arcs manually annotated at 2m resolution from nuScenes with the help of nuscenes-devkit. LaneSlice instances also represent lane widths.

[0166] The given center point whose width is to be calculated is projected to the left and right borders. The projected points are those points on the border that have the minimum Euclidean distance to the given center point. The width is given by the distance between the projected points. StopArea.

[0167] Stopping areas are a very important concept for motion prediction, since by definition they are areas where cars tend to stop. There are several causes for such areas, and each of them is modeled as a subclass of the parent class StopArea. These causes include stop signs, yield signs, oncoming vehicles when wanting to turn left, crosswalks, and traffic lights. causesStopAt links a cause entity to its associated StopArea.

[0168] Equation (13) provides the definition of the concept StopArea.

[0169] StopArea≡PedCrossingStopArea∪TrafficLightStopArea∪

[0170] YieldStopArea∪StopSignArea∪TurnStopArea(13)

[0171] StopArea instances are generated by nuScenes STOP LINE records.

[0172] TrafficLight. Their physical appearance is represented using the categorical data attribute hasapricLightType to distinguish between horizontally and vertically stacked lights. In addition, the lights are located at a specific position and facing a specific direction, which is represented via trafficLightHasPose pointing to a specific Pose instance. The dynamic state of traffic lights (light color) is not modeled because this information is not available in the nuScenes dataset.

[0173] Equation (14) provides the definition of the concept TrafficLight.

[0174]

[0175] TrafficLight instances are generated from nuScenes TRAFFIC LIGHT records.

[0176] PedCrossing. This is where pedestrians can legally cross the street. Two walkways connected by a crossing are represented using the connectsWalkways relationship.

[0177] Equation (15) provides the definition of the concept PedCrossing.

[0178] PedCrossing ≡≤ 2connectsWalkways.Walkway (15)

[0179] Visual inspection of several crosswalks and walkways in the nuScenes dataset shows that they are not necessarily touching, but they are always close.

[0180] As a heuristic, walkways within 5 meters of the crosswalk are considered. Our algorithm selects the two walkways with the smallest distance. To check the correctness of the implementation, a subset of the generated triplets is visualized and manually verified. Walkway, CarparkArea.

[0181] Walkways are modeled with the concept of the same name. CarparkArea is any area where a car can be parked, either in an actual carpark or on the side of the road. To explicitly represent the proximity between adjacent parts of the road, isNextTo exists. carparkAreaIsNextTo represents a lane adjacent to a carpark, and similarly, walkwayIsNextTo represents the adjacency between a walkway and a lane.

[0182] Equations (16) and (17) provide the definitions of the concepts WalkWay and CarparkArea.

[0183]

[0184] The isNextTo relationships between walkways, driveways, and parking lots are generated for those entity pairs that are within 4m.

[0185] This heuristic threshold was chosen after visualizing several lanes, parking lots, and walkways and their proximity. In this way, a clear spatial relationship is established between adjacent road surfaces. Road blocks 627 group adjacent lanes that travel in the same direction. If one block to another contains lanes that are connected to each other, there is a hasNextRoadBlock edge.

[0186] Thus, the roadblock 627 connectivity models any potential future areas where the car could go, taking into account possible lane switches, whereas lane connectivity does not take into account possible lane switches. Additionally, a hasOpposingRoadBlock relationship is introduced. If two roadblocks 627 are parallel to each other on the same road, carrying traffic in opposite directions, then the hasOpposingRoadBlock relationship exists between the two roadblocks 627. This extends the spatial connectivity in the graph, such as the isNextTo property, to make spatial relationships explicit and intuitive for humans to see.

[0187] Equation (18) provides the definition of the concept RoadBlock.

[0188]

[0189] Roadblock instances are computed by grouping adjacent lanes, and connectivity between roadblocks 627 is determined by lane connectivity. Instances could not be generated from the nuScenes ROAD BLOCK 627 records because they contained malformed shapes on two of the four maps, as raised in the GitHub issue. Intersection. This is where multiple lanes intersect.

[0190] A typical path through an intersection is characterized by lane connectors across the intersection. Lane 519, 547, 549, 551 enters the intersection and lane 519, 547, 549, 551 leaves the intersection if and only if a car is allowed to travel through the intersection from lane 519, 547, 549, 551 entering the intersection to lane 519, 547, 549, 551 leaving the intersection. isConnectorOnRoadSegment associates an intersection with a lane connector on said intersection.

[0191] Equation (19) provides the definition of the concept Intersection.

[0192]

[0193] An intersection instance is generated by a ROAD SEGMENT record. An explicit spatial link between an intersection and a lane connector is computed by checking if the lane connector overlaps the intersection. hasShape. In order to model the exact position, shape and size of all the above map elements, a hasShape relationship is introduced for each element.

[0194] Each shape is represented by a subclass of the GeoSPARQL Simple Features (prefix sf) ontology concept sf:Geometry. For example, it includes sf:Polygon, which is used to model polygonal structures such as walkways, lanes or intersections. The data properties of a geometry store its exact shape in nuScenes (x, y) coordinates, but also in GPS coordinates, representing a real location on the earth.

[0195] This enables integration with other geographic data sources and geospatial analysis. isOn, AreaElement. To create a connection between the agent and the map, the isOn relationship is introduced. AreaElement is defined as the superclass for all map elements that occupy an area, i.e. have a sf:Polygon geometry. isOn links the SceneParticipant to the map object it is currently on.

[0196] Equation (20) provides the definition of the concept AreaElement.

[0197] AreaElement≡Walkway∪CarparkArea∪Lane∪LaneSnippet∪

[0198] RoadBlock∪StopArea∪PedCrossing∪Intersection(20)

[0199] The nSKG graph representation is prepared as an easily loadable dataset. It is provided in the format of a PyTorch Geometric (PyG) graph. PyTorch Geometric (PyG) is one of the most widely used graph neural network libraries, so the dataset is prepared in a PyG compatible format and is easily loaded by the PyG data loader. Formally, the heterogeneous graph G = (V, E, τ, φ) has nodes v∈V of node type τ(v) and edges (u, v)∈E of edge type φ(u, v). The edges are directed as they are based on the properties of the knowledge graph.

[0200] Each example i in the constructed dataset is a pair of (x i ,y i )∈(G,R 12 ), where x i is a scene graph with trajectory information from the past two seconds, local maps, and target identifiers, y i is the ground truth future trajectory of the target. This makes our dataset a graph regression task.

[0201] The constraints for the past 2 seconds and the future 6 seconds (sampled at 2Hz) are preserved by nuScenes so that any results on our new graph dataset can be compared with the results on the nuScenes original data. The training, validation, and test splits from nuScenes are also preserved.

[0202] The coordinate system used is an important consideration because the correct choice of coordinate system can achieve data-centric inductive bias, that is, translation and rotation invariance. Inductive bias is widely believed to be the key to achieving good generalization in deep learning.

[0203] Coordinates in the knowledge graph (and nuScenes) are initially in the global coordinate system. These coordinates are transformed into local, scene-graph-specific coordinates for each scene graph, with the origin at the location of the target agent and the positive x-axis pointing in the direction facing the target. Specifically, let p target and R target Correspondingly, it becomes the global position (vector) and orientation (rotation matrix) of the target vehicle in the scene graph g. Let p global , R global Correspondingly, are any global position and global orientation. They are represented in the local coordinate system by:

[0204] Equations (21) and (22) provide the concept p local and R local Definition of .

[0205] p local = R -1 target (pglobal -p target ) (twenty one)

[0206] R local = R -1 target R global (twenty two)

[0207] Where R -1 target is the rotation matrix R target The inverse matrix of . In this way, the coordinates of all entities in g can be transformed into the local coordinate system. The prediction automatically becomes translation-invariant and rotation-invariant because any translation and rotation are removed in the transformation.

[0208] In all examples i, the target is located at the origin, along the positive x-axis. Empirically, this transformation improves trajectory prediction performance.

[0209] The trajectory information in the scene graph contains Sequence 537, Scene, SceneParticipant and Participant nodes and the semantic relationship between SceneParticipants. The object attributes between them become heterogeneous edges in the knowledge graph.

[0210] Their data attributes are converted into node features. A SPARQL query is used to retrieve the Scene instances and the relevant agents within them for the past two seconds. Relevant agents are those that could potentially affect the motion of the target vehicle, defined as those on the relevant extraction map described next. This no longer considers scene actors that are already behind the target agent, an improvement over previous work.

[0211] In addition to trajectory information, the scene graph contains rich map information that is modeled in our ontology. However, including the entire city map would be counterproductive and would make the graph unnecessarily large. The larger the graph, the more long-range dependencies may arise, which poses a problem for state-of-the-art graph neural networks. Only those parts of the map that are potential paths to the goal are considered. To extract these parts from the knowledge graph, the goal is mapped to the roadblock 627 where it is located and follows the hasNextRoadBlock edge to four blocks into the future.

[0212] Four roadblocks 627 were chosen as the relevant length after analyzing the distance traveled by the agent in 6 seconds in nuScenes. Roadblocks 627 are typically lane segment lengths or longer, and lane connectors are around 5-10m, so in most cases, four roadblocks 627 cover the future trajectory since the agent typically does not travel more than 120 meters, as revealed by our analysis. Adding more to the future than necessary would make the graph larger than necessary, hurting the performance of current graph neural networks. The surrounding map entities of potential paths, walkways, parking lots, crosswalks, and stopping areas, etc. are extracted via explicit spatial relationships modeled in the ontology, as described in the previous section.

[0213] These explicit spatial relations are also preserved in the heterogeneous scene graph and, like all other object properties, are converted into heterogeneous edges as described previously. The connection between the extracted trajectory information and the map is established via the isOn relationship, just like in the knowledge graph.

[0214] The constructed graph dataset contains over 40,000 scene graphs and ground truth future trajectories. Each scene graph contains an average of 1,000 to 2,000 nodes. Graph neural networks trained on these scene graphs are able to access more information than any previous trajectory prediction method. It is possible that such representations through appropriate graph neural networks will outperform current state-of-the-art methods, as competing methods cannot access all the information important for trajectory prediction.

[0215] Figure 5 A flow chart of a method 100 for generating a knowledge graph 400 for traffic movement prediction according to one embodiment is shown.

[0216] In order to generate the knowledge graph 400 for traffic movement prediction in the first method step 101, environmental sensor data 503 of at least one environmental sensor 505 of the host vehicle 501 are received. The environmental sensor data 503 thus represent the environment of the host vehicle 501 and include information about at least one traffic participant 507 located in the environment of the host vehicle 501.

[0217] The environmental sensor 505 may be, for example, a camera, a radar sensor, an ultrasonic sensor, or any other common environmental sensor used in autonomous driving.

[0218] In a second method step 103, map data 509 are received from an electronic road map 511. The map data 509 represent a road network 513 in the environment of the host vehicle 501 and include information about at least one movement trajectory 515 on which the traffic participant 507 is located.

[0219] Traffic participants 507 may be static objects, moving objects, people, animals, vehicles, cars, trucks, trams, motorcycles, bicycles, obstacles, traffic cones or any other objects that can be detected via the above-mentioned environmental sensors 505 in normal traffic scenarios.

[0220] Depending on the type of traffic participant 507, the movement trajectory 515 can be a road 517, a lane 519, 547, 549, 551, an intersection, an underpass, a bridge, a highway, a highway entrance, a highway exit, a roundabout 521, a parking lot, a parking lot, a bicycle lane, a tramway, a crosswalk 523, a sidewalk, or any other infrastructure in the environment of the vehicle 501 that allows the movement of the traffic participant 507.

[0221] In a further method step 105 , information about at least one road user 507 and information about movement trajectory 515 are extracted from surrounding sensor data 503 and map data 509 .

[0222] In a further method step 109 , time stamps t1 , t2 of surroundings sensor data 503 , 509 are determined.

[0223] In a further method step 111 , the environmental sensor data 503 and the map data 509 are organized in scenes 533 , 535 and sequences 537 . A scene 533 , 535 comprises information about the environmental sensor data 503 and the map data 509 for one time stamp t1 , t2 . A sequence 537 is a series of consecutive scenes 533 .

[0224] In a further method step 113 , the nodes 401 of the knowledge graph 400 are organized with respect to the scenes 533 , 535 and sequences 537 of the environmental sensor data 503 and the map data 509 .

[0225] In a further method step 107, a knowledge graph 400 comprising a plurality of nodes 401 and edges 403 is generated based on the finally extracted information about the traffic participants 507 and the movement trajectories 515 of the map data 509 and the environmental sensor data 503. The knowledge graph 400 thus comprises at least a node 401 presenting information about the traffic participants 507 and a node 401 representing information about the movement trajectories 515.

[0226] According to one embodiment, further information about the motion trajectory 515 is included in the knowledge graph 400 .

[0227] Further information may include road geometry, lane geometry, lane dividers, lane boundaries, lane connectors, intersections, stopping areas, traffic signals, traffic signs, traffic regulations, road conditions, slope values, crosswalks, parking areas, road sections, roadblocks 627 or any other feature of the possible movement trajectory 515 that may appear in a normal traffic scenario and that may influence the movement of the respective traffic participant 507.

[0228] In addition to the above characteristics of traffic participant 507, other characteristics regarding the relative position of traffic participant 507 to other traffic participants or to host vehicle 501 may also be included. The relative position may include a longitudinal relative position 553, a lateral relative position 555, and / or an intersection relative position 557.

[0229] According to another embodiment, the information of the map data 509 considered in the knowledge graph 400 is limited to the area of ​​the possible path of the traffic participant 507. Information about areas in the road network 513 that the traffic participant 507 cannot reach from its current position is not introduced into the knowledge graph 400. This reduces the complexity of the knowledge graph 400 and simplifies the generation of the knowledge graph 400 and the computing power and storage capacity required to generate and operate the knowledge graph 400.

[0230] According to one embodiment, the method 100 for generating a knowledge graph 400 for traffic movement prediction is performed during an operating mode of the host vehicle 501 , in particular during a driving mode.

[0231] Figure 6 A flow chart of a method 200 for traffic movement prediction according to one embodiment is shown.

[0232] In order to perform the movement prediction in the first method step 201 , a knowledge graph 400 for traffic movement prediction is generated by performing the method 100 for generating a knowledge graph 400 for traffic movement prediction according to the above-described embodiment.

[0233] In a further method step 203 , a future movement of at least one road user 507 in the surroundings of host vehicle 501 is predicted by movement prediction module 539 based on knowledge graph 400 .

[0234] According to one embodiment, the movement prediction module 539 includes a trained artificial intelligence capable of predicting the movement of the traffic participant 507 based on the information of the knowledge graph 400 .

[0235] The trained artificial intelligence may include a trained graph network.

[0236] Figure 7 A flow chart of a method 300 for controlling a host vehicle 501 according to one embodiment is shown.

[0237] In order to control host vehicle 501 in first method step 301 , a future movement of at least one traffic participant 507 in the environment of host vehicle 501 is predicted by carrying out method 200 for traffic movement prediction according to the above-described exemplary embodiment.

[0238] In a further method step 303 , at least one control function of host vehicle 501 is executed based on the predicted movement of at least one road user 507 .

[0239] The control functions of the vehicle 501 may include a steering function and / or an acceleration or deceleration function and / or a deceleration function of the vehicle 501 .

[0240] Figure 8 A schematic diagram of a computer program product 700 comprising instructions which, when executed by a data processing unit, cause the program to perform the method 100 for generating a knowledge graph for traffic movement prediction and / or the method 200 for traffic movement prediction and / or the method for controlling the host vehicle 501 .

[0241] In the illustrated embodiment, the computer program product 700 is stored on a computing unit or storage medium 701. The storage medium 701 may be any storage medium known in the art.

Claims

1. A computer-implemented method (100) for generating a knowledge graph (400) for traffic movement prediction, the method comprising: Receiving (101) environmental sensor data (503) of at least one environmental sensor (505) of a host vehicle (501), wherein the environmental sensor data (503) represent the environment of the host vehicle (501) and include information about at least one traffic participant (507) located in the environment of the host vehicle (501); Receiving (103) map data (509) from an electronic road map (511), wherein the map data (509) represent a road network (513) in the environment of the host vehicle (501) and include information about at least one movement trajectory (515) of the traffic participant (507); extracting (105) information about the at least one traffic participant (507) from the environmental sensor data (503), and extracting information about a movement trajectory (515) of the traffic participant (507) from the map data (509); and A knowledge graph (400) of the road network (513) in the environment of the host vehicle (501) is generated (107) based on the map data (509) and / or the environmental sensor data (503), the knowledge graph comprising nodes (401) and edges (403), wherein the knowledge graph (400) comprises at least one node (401) representing the traffic participant (507) and at least one node (401) representing the lane in which the traffic participant (507) is located.

2. The method (100) according to claim 1, wherein: The movement trajectory (515) on which the traffic participant (507) is located is at least one of the following list: road (517) and / or lane (519, 547, 549, 551), intersection, underpass, bridge, highway, highway entrance, highway exit, roundabout (521), parking lot, parking lot, bicycle lane, tramway, crosswalk (523), sidewalk.

3. The method (100) according to claim 1 or 2, wherein: The map data (509) of the electronic road map (511) include further information about further features of the movement trajectory (515) on which the traffic participant (507) is located, and wherein at least one further feature of the movement trajectory (515) of the traffic participant (507) is integrated into the knowledge graph (400) via at least one further node.

4. The method (100) according to claim 1 or 2, wherein: The environmental sensor data (503) also include further information about at least one further feature of the traffic participant (507), and wherein the at least one further feature of the traffic participant (507) is integrated into the knowledge graph (400) via at least one further node.

5. The method (100) according to claim 3 or 4, wherein: The node (401) and further nodes of the knowledge graph (400) are organized into classes and subclasses.

6. The method (100) according to any one of claims 3 to 5, wherein: The further feature of the motion trajectory (515) is at least one of the following list, the list including: road geometry, lane geometry, lane divider (525), lane boundary (527), lane connector, intersection, stop area, traffic signal, traffic sign, traffic rules, road condition, slope value, crosswalk, parking area, road segment, roadblock (627).

7. The method (100) according to any one of claims 4 to 6, wherein: The further characteristic of the traffic participant (507) is at least one of the following list, the list comprising: static object, moving object, person (529), animal, vehicle, car (531), truck, tram, motorcycle, bicycle, obstacle, traffic cone, relative position (553, 555, 557) with at least one other traffic participant (543) and / or with the vehicle (501).

8. The method (100) according to any one of the preceding claims, further comprising: determining (109) timestamps (t1, t2) of the environmental sensor data (503) and the map data (509); Organizing (111) the environmental sensor data (503) and the map data (509) into scenes (533, 535) and sequences (537), wherein the scenes (533, 535) include information of the environmental sensor data (503) and the map data (509) for a timestamp (t1, t2), and wherein the sequence (537) is a series of consecutive scenes (533); and The nodes (401) of the knowledge graph (400) are organized (113) relative to the scenes (533, 535) and the sequences (537) of the environmental sensor data (503) and the map data (509).

9. The method (100) according to any one of the preceding claims, wherein: The information of the map data (509) taken into account when generating the knowledge graph (400) is limited to the area of ​​possible paths of the road users (507).

10. The method (100) according to any one of the preceding claims, wherein: The method (100) is performed during a driving operation of the host vehicle (501).

11. A computer-implemented method (200) for traffic movement prediction, the method comprising: Generating (201) a knowledge graph (400) for traffic movement prediction by executing the method (100) for generating a knowledge graph (400) for traffic movement prediction according to any one of the preceding claims 1-10; The future movement of at least one traffic participant (507) in the environment of the host vehicle (501) is predicted (203) based on the knowledge graph (400) by a movement prediction module (539).

12. The method (200) of claim 11, wherein: The movement prediction module (539) includes a trained artificial intelligence that is capable of predicting the movement of the traffic participant (507) based on the information of the knowledge graph (400).

13. A computer-implemented method (300) for controlling a host vehicle (501), the method comprising: Predicting (301) the future movement of at least one traffic participant (507) in the environment of the host vehicle (501) by executing the method (200) for traffic movement prediction according to claim 11 or 12; and At least one control function of the host vehicle (501) is performed (303) based on the predicted movement of the at least one road user (507).

14. A computing unit (541) configured to execute the method (100) for generating a knowledge graph (400) for traffic movement prediction according to any one of the preceding claims 1-10 and / or the method (200) for traffic movement prediction according to the preceding claim 11 or 12 and / or the method for controlling the vehicle (501) according to claim 13.

15. A computer program product (700) comprising instructions, which, when executed by a data processing unit, cause the data processing unit to execute a method (100) for generating a knowledge graph (400) for traffic movement prediction according to any one of the preceding claims 1-10 and / or a method (200) for traffic movement prediction according to the preceding claims 11 or 12 and / or a method for controlling a vehicle (501) according to claim 13.

Images