Methods for generating a knowledge diagram for traffic movement prediction, methods for traffic movement prediction, and methods for controlling an ego vehicle.

Abstract

The method relates to a computer-implemented method (100) for generating a knowledge diagram (400) for traffic movement prediction, comprising: Receiving (101) environmental sensor data (503) from at least one environmental sensor (505) of an ego vehicle (501); Receiving (103) map data (509) from an electronic road map (511); Extracting (105) information about the at least one road user (507) from the environmental sensor data (503) and extracting information about the movement lane (515) on which the road user (507) is located from the map data (509);and generating (107) a knowledge diagram (400) of the road network (513) in the vicinity of the ego-vehicle (501), comprising nodes (401) and edges (403), based on the map data (509) and / or the environmental sensor data (503), wherein the knowledge diagram (400) comprises at least one node (401) representing the road user (507) and at least one node (401) representing the lane on which the road user (507) is located. The invention further relates to a method (200) for predicting traffic movement. The invention further relates to a method (300) for controlling an ego-vehicle (501);

Description

The present invention relates to a method for generating a knowledge diagram for traffic motion prediction. The invention further relates to a method for traffic movement prediction. The invention further relates to a method for controlling an ego vehiclePrior ArtMotion prediction in traffic scenes involves the accurate prediction of the behavior of surrounding road users or other moving objects. To intuitively solve this task, several information sources must be considered, including the travel path of vehicles, the road topology, lane separators, pedestrian cross-ties, and traffic rules. Although previous studies have demonstrated the potential of using heterogeneous context to improve motion prediction, highly-modern deep learning approaches still rely on a limited subset of this information. This is primarily due to the limited availability of comprehensive representations of the context information available in prominent datasets.Traffic movement prediction is a key part of autonomous driving because it allows the autonomous vehicle to predict other road users' movement and avoid dangerous situations that could lead to collisions. Machine learning, in particular deep learning approaches, have proven very successful in developing motion predictions and frequently achieve high accuracy. Readily usable data sets were of decisive importance for the progress in machine learning. MNIST, COCO and ImageNet were critical to the progress in the computer vision, GLUE and SQuAD areas for natural language understanding and training environments, MuJoCo and OpenAl Gym areas for reinforcement learning, just to name a few. In recent years, however, the limits of deep learning models have been widely studied.In particular, their lack of robustness and clarity, as well as their inability to generalize them to new domains. One hypothesis why modern deep learning models lack robustness is that they are purely sub-symbolic and statistical and learn correlations between input features and target variables, rather than obtaining causal structured understanding of a task.People think of driving not at the level of individual pixels, but at the level of objects, perceived intentions, and spatial and semantic relationships. A person has an inherent prior knowledge, for example, the realization that no road user can achieve an acceleration above a specific threshold value. High-level structured information (knowledge) is typically missing when deep learning models train end-to-end scenarios from raw data. Moreover, for real applications of motion prediction algorithms, it is important to take into account safety aspects. ISO26262, the international standard for the functional safety of road vehicles, must be fulfilled. In this case, it is required that the behavior of components is fully specified and validated.It is an object of this invention to provide an improved method of generating a traffic motion prediction knowledge diagram and an improved traffic motion prediction method and an improved method of controlling an ego vehicle.This object is achieved by the methods of the independent claims. Further embodiments are the subject of the dependent claims.According to one aspect of the invention, there is provided a computer implemented method for generating a traffic motion prediction knowledge diagram comprising:receiving environment sensor data of at least one environment sensor of an ego vehicle, the environment sensor data representing the environment of the ego vehicle and comprising information about at least one road user located in the environment of the ego vehicle;receiving map data of an electronic road map, the map data representing a road network in the vicinity of the ego vehicle and comprising information about at least one movement lane in which the road user is located;extracting the information about the at least one road user from the environment sensor data and extracting the information about the road and / or lane on which the road user is located from the map data; andgenerating a knowledge diagram of the road network in the environment of the ego vehicle, comprising nodes and edges, based on the map data and / or the environment sensor data, wherein the knowledge diagram comprises at least one node representing the road user and at least one node representing the lane on which the road user is located.This can achieve the technical advantage that an improved method for generating a knowledge diagram for traffic motion prediction can be provided. Environment sensor data of at least one environment sensor of an ego vehicle are taken into account as information for the knowledge diagram. The environment sensor data represent the environment of the ego vehicle and comprise information about at least one road user located in the environment of the ego vehicle. Information about this at least one road user located in the environment of the ego vehicle can be extracted via the environment sensor data.Furthermore, map data of an electronic road map is also received, which represents a road network in the environment of the ego vehicle. Information relating to at least one movement lane of the road user who is located in the environment of the ego vehicle can be extracted via the map data.Information regarding individual features of the road user can be taken into account via the environment sensor data for generating the knowledge diagram. Information regarding features of the movement track of the road user can be taken into account via the map data of the electronic road map when generating the knowledge diagram.In the generation of the knowledge diagram, the information extracted from the environment sensor data and the information extracted from the map data are included in the knowledge diagram by means of nodes and / or edges of the knowledge diagram.As a result, a knowledge diagram can be generated which comprises information about individual features of the road users located in the environment of the ego vehicle and information about individual features of the movement lanes of these road users. This enables the generation of a knowledge diagram with precise information about road users and respective movement paths of these road users in the environment of the ego vehicle.Based on this knowledge diagram, a precise and reliable traffic movement prediction of the future movement of road users located in the environment of the ego vehicle is possible.In the sense of this application, the travel path of the road users is defined as a distinguishable area in the road network. The movement lane is part of the infrastructure in the environment of the ego vehicle on which the respective road user is located and which enables the road user to move in the future.According to one embodiment, the movement lane in which the road user is located is at least one from the following list: a road and / or lane, intersection, underpass, bridge, freeway, freeway entrance, freeway exit, a circular traffic, a parking bay, a parking lot, bicycle path, highway track, a pedestrian crossing, sidewalk.This can achieve the technical advantage that, when generating the knowledge diagram, a multiplicity of different movement paths for different road users in the environment of the ego vehicle can be taken into account. In this way, a knowledge diagram can be generated for a plurality of different road users, which are located at different locations in the environment of the ego vehicle. This enables precise and flexible traffic motion prediction based on a corresponding knowledge diagram applicable to many different traffic scenarios.According to one embodiment, the map data of the electronic road map comprise further information about further features of the movement track on which the road user is located, and wherein the at least one further feature of the movement track of the road user is integrated into the knowledge diagram via at least one further node.This can achieve the technical advantage that additional information regarding the movement track can be introduced into the knowledge diagram via the further features of the movement track of the road users. This additional information regarding the motion track enables a more accurate prediction of the traffic motion based on a respective knowledge diagram.According to one specific embodiment, the surroundings sensor data also include further information regarding at least one further feature of the road user, and the at least one further feature of the road user being integrated into the knowledge diagram via at least one further node.This can achieve the technical advantage that further information regarding further individual features of the road users can be introduced into the knowledge diagram. On the basis of this further information, a more detailed description of the road user can be introduced into the knowledge diagram. This makes it possible to achieve a more accurate movement prediction of the future movement of the respective road user on the basis of the respective knowledge diagram.According to one embodiment, the nodes and other nodes of the knowledge diagram are organized into classes and subclasses.This can achieve the technical advantage that organization of the information within the knowledge diagram is achieved by organization of the nodes of the knowledge diagram into classes and subclasses.According to one specific embodiment, the further features of the motion lane are at least one from the following list, including: road geometries, lane geometries, lane dividers, lane boundaries, lane connections, intersections, holding regions, traffic signals, traffic signs, traffic rules, road conditions, gradient values, pedestrian crossings, parking space areas, road sections, road blocks.This can achieve the technical advantage that the knowledge diagram can be generated for a plurality of different road users who are located on a plurality of different movement lanes in the environment of the ego vehicle. This enables flexible and likely applicable traffic motion prediction based on a corresponding knowledge diagram.According to one embodiment, the further features of the road user are at least one from the following list, which comprises: a static object, a moving object, a human, an animal, a vehicle, a car, a truck, a roadway, a motorcycle, a bicycle, a barrier, a traffic cone, a relative position to at least one further road user and / or to the ego vehicle.This can achieve the technical advantage that a plurality of different road user types can be taken into account in the generation of the respective knowledge diagram. This enables widely applicable traffic motion prediction based on a corresponding knowledge diagram.According to an embodiment, the method further comprises:determining a time stamp of the environment sensor data and the map data;organizing the environment sensor data and the map data into scenes and sequences, wherein a scene comprises the information of the environment sensor data and the map data for a time stamp, and wherein a sequence is a series of successive scenes; andorganizing the nodes of the knowledge diagram with respect to the scenes and sequences of the environment sensor data and the map data.This can achieve the technical advantage that a time instance can be introduced into the knowledge diagram by organization of the nodes of the knowledge diagram taking account of scenes and sequences of the environment sensor data and the map data. This enables the use of the knowledge diagram for traffic movement prediction based on time-dependent data, as occur during continuously changing traffic scenarios during the operation of the ego vehicle. As a result, the respective knowledge diagram can be used for traffic movement prediction and can be applied during operation of the ego vehicle.According to one embodiment, the information of the map data taken into account in the generation of the knowledge diagram is limited to a region of a possible path of the road user.This can achieve the technical advantage that the knowledge diagram can be reduced in size and can be reduced to the information essential for traffic movement prediction. This enables a fast and efficient prediction of the traffic movement.According to one specific embodiment, the method is carried out during a driving mode of the ego vehicle.This can achieve the technical advantage that the knowledge diagram can be generated and updated during the operation of the ego vehicle. This enables a precise prediction of the traffic movement.According to one aspect, there is provided a computer implemented method for traffic motion prediction, comprising:generating a traffic motion prediction knowledge diagram by executing the traffic motion prediction knowledge diagram generating method according to any one of the previous embodiments; andpredicting a future movement of at least one road user who is located in a surrounding area of an ego vehicle based on the knowledge diagram by a movement prediction module.This can achieve the technical advantage that an improved method for traffic movement prediction can be provided. The method for controlling the ego vehicle uses the method for generating a knowledge diagram for traffic motion prediction with the above-mentioned technical advantages.According to an embodiment, the motion prediction module comprises a trained artificial intelligence capable of predicting the motion of the road user based on information of the knowledge diagram.This can achieve the technical advantage that a robust and efficient prediction module can be provided.According to an embodiment, the artificial intelligence comprises a graphical neural network.This can achieve the technical advantage that a robust and efficient prediction module can be provided.According to one aspect, there is provided a computer-implemented method for controlling an ego vehicle, comprising:predicting a future movement of at least one road user who is located in a surrounding area of the ego vehicle by carrying out the method for traffic movement prediction according to one of the preceding embodiments; andexecuting at least one control function of the ego vehicle based on the predicted movement of the at least one road user.This can achieve the technical advantage that an improved method for controlling an ego vehicle can be provided. The method for controlling the ego vehicle uses the method for traffic movement prediction with the technical advantages mentioned above.According to one aspect, a computing unit is configured to execute the method for generating a knowledge diagram for traffic movement prediction according to one of the preceding embodiments and / or the method for traffic movement prediction according to one of the preceding embodiments and / or the method for controlling an ego vehicle.According to one aspect, there is provided a computer program product comprising instructions which, when the program is executed by a data processing unit, cause the data processing unit to execute the method for generating a knowledge diagram for traffic motion prediction according to any of the preceding embodiments and / or the method for traffic motion prediction according to any of the preceding embodiments and / or the method for controlling an ego vehicle.Exemplary embodiments of the invention are described with reference to the following figures. The figures show: FIG. 1 is a schematic illustration of a system for generating a knowledge diagram for traffic motion prediction according to an embodiment; FIG. 2 shows a further schematic illustration of a system for generating a knowledge diagram for traffic movement prediction according to a further embodiment; FIG. 3 is another schematic illustration of a system for generating a knowledge diagram for traffic motion prediction according to another embodiment; FIG. 4 is a schematic diagram of a knowledge diagram for traffic motion prediction according to an embodiment; FIG. 5 is a flow diagram of a method for generating a knowledge diagram for traffic motion prediction according to an embodiment; FIG. 6 is a flow diagram of a method for traffic motion prediction according to an embodiment; FIG. 7 shows a flow chart of a method for controlling an ego vehicle according to an exemplary embodiment; and FIG. 8 is a schematic illustration of a computer program product.FIG. 1 shows a schematic representation of a system 500 for generating a knowledge diagram 400 for traffic motion prediction according to an embodiment.FIG. 1 shows two traffic scenarios represented in a first scene 533 and a second scene 535 shown in graphics a) and b). The first and second scenes 533, 535 have a time stamp t 1, t 2 at which the respective traffic scenarios were observed.In the two traffic scenarios, an ego vehicle 501 is shown traveling on a road 517 that includes a first lane 519 and a second lane 547. In the two traffic scenarios illustrated, a further road user 507 is located in a surrounding area of the ego vehicle 501. In the embodiment shown, the further road user 507 is a vehicle 565, in particular a car 531. The road user 507 moves on a movement track 515.The movement track 515 is defined by an infrastructure in the environment of the ego vehicle, which infrastructure enables a future movement of the road user 507. In the example shown, the travel lane 515 is given by the road 517, and more specifically by one of the lanes 519, 547.In the two scenes 533, 535, a travel scenario of the ego vehicle 501 and of the further road user 507 is shown. The two scenes 533, 535 describe two successive traffic scenarios, wherein the road user 507 carries out a lane change from scene 533 to scene 535 and changes its position from the first lane 519 to the second lane 547. The two traffic scenarios shown are just one example and are not intended to limit the scope of the present invention.In the specific embodiment shown, host vehicle 501 includes a computing unit 541 and at least one surroundings sensor 505. Via environment sensor 505, ego vehicle 501 may collect environment sensor data 503 that represents the environment of ego vehicle 501. Environment sensor 505 may be, for example, a camera sensor, an ultrasonic sensor, a radar sensor, or any environment sensor known in the field of automated driving. Based on surroundings sensor data 503, ego vehicle 501 is able to detect other road users 507 situated in the surroundings of ego vehicle 501.Furthermore, the ego vehicle 501 has access to an electronic road map 511. The electronic road map 511 represents a road network 513 having a plurality of roads, lanes, and other features of the traffic infrastructure, and provides detailed information about the roads, lanes, and infrastructure surrounding the ego vehicle 501.By using the arithmetic unit 541, the map data 509 and the environment sensor data 503, the host vehicle is able to execute the method for generating a knowledge diagram for traffic motion prediction according to the present invention.For this purpose, the ego vehicle 501 detects the road user 507 in the environment of the ego vehicle 501 and the respective movement lane 515 via the environment sensor data 503.Via the map data 509 of the electronic road map 511, further information about the detected and / or identified movement lane 514 of the road user 507 is received.Based on the environment sensor data 503 and the map data 509, the movement lane 514 of the road user 507 can be identified, for example, as road 517 and / or lane 519, 547, intersection, underpass, bridge, freeway entrance, freeway exit, circular traffic, parking bay, parking lot.Based on the environment sensor data, the road user 507 may be further identified as a static object, a moving object, a human, an animal, a vehicle 565, a car 531, a truck, a roadway, a motorcycle, a bicycle, a barrier or a traffic cone.Based on the respective type of road user 507, the movement track 515 can also be identified as a bicycle path, road track, pedestrian crossing or sidewalk, or any other possible movement track 515.From the map data 509, further individual information regarding the previously identified movement track 515 can be taken into account. The further features may include road geometries, lane geometries, lane dividers 525, lane boundaries 527, lane connections, intersections, stopping areas, traffic signals, traffic signs, traffic rules, road conditions, slope values, pedestrian crossings, parking lot areas, road segments, road blocks, or other features that represent the infrastructure of a traffic scenario.To generate the knowledge diagram based on the information from map data 509 and environment sensor data 503, the above-mentioned information regarding motion track 515 and participant track 507 is extracted from environment sensor data 503 and map data 509.Based on this extracted information regarding the motion lane 515 and the road user 507, the knowledge diagram is generated, and the extracted information about a plurality of nodes and edges connecting these nodes is included in the knowledge diagram.According to one embodiment of the invention, the respective knowledge diagram is organized in the above-mentioned scenes 533, 535, which respectively describe subsequent traffic scenarios that take place at different time stamps t 1, t 2. In this way, the knowledge diagram is organized in a time-dependent form, wherein for each scene 533, 535 and the respective time stamp t1, t2, the above-mentioned information is included in the knowledge diagram with corresponding nodes and edges.In FIG. 1, the two scenes 533, 535 shown define a sequence 537. Sequence 537 describes the sequential dependence of the displayed scenes 533, 535. The sequence information can also be included as additional information in the knowledge diagram.Lane dividers 525 and lane boundaries 527 are shown in the two graphs a), b) in addition to the road 517 and lanes 519, 547.In the illustrated embodiment, the ego vehicle 501 further includes a motion prediction module 539 installed on the computing unit 541. Via the motion prediction unit 531, the subject vehicle 501 is capable of executing the traffic motion prediction method according to the present invention.To carry out the traffic movement prediction, the ego vehicle 501 acquires the environment sensor data 503 during operation in order to identify and identify the road user 507 and the respective movement lanes 515 of the road user 507 located in the environment of the ego vehicle 501.Furthermore, information about the road network 513 within the environment of the ego vehicle 501 is extracted from the map data 509 of the electronic road map 511. Based on the information extracted from map data 509 and surroundings sensor data 503, ego vehicle 501 generates the above-mentioned knowledge diagram. This generation of the knowledge diagram can be carried out in the operating mode, in particular the driving mode, of the ego vehicle 501.Based on the information of the knowledge diagram, the traffic prediction module 539 is able to predict a future movement of the road users 507 located and identified in the environment of the ego vehicle 501. To this end, the traffic prediction module 539 may be configured as trained artificial intelligence capable of performing motion prediction based on the information of the above-mentioned knowledge diagram. To this end, the motion prediction module 539 may be designed as a diagram network capable of performing the motion prediction.Via the arithmetic unit 541, the ego vehicle 501 is also able to execute the method for controlling the ego vehicle 501 according to the present invention. For this purpose, the ego vehicle 501 first carries out the method for traffic movement prediction and predicts a future movement of the further road users 507 located and identified in the environment of the ego vehicle 501. Based on the predicted future movement of the identified road users 507, the ego vehicle 501 performs at least one control function. The control function can be, for example, an acceleration or deceleration and / or a steering function.FIG. 2 shows a further schematic illustration of a system 500 for generating a knowledge diagram 400 for traffic movement prediction according to a further embodiment.FIG. 2 shows another traffic scenario as another example of the present invention. In the traffic scenario shown, a further road user 507 is located in the environment of the ego vehicle 501. In the current scenario, the further road user 507 is a human 529. The human 529 moves on its movement track 515. The motion lane 515 is, in the present example, a pedestrian crossing 523 that crosses the lane 519 of the road 517 on which the ego vehicle 501 is located. Apart from the differences mentioned, the embodiment shown in FIG. 2 is based on the embodiment of FIG. 1 and comprises all features shown in FIG. 1.FIG. 3 shows a further schematic illustration of a system 500 for generating a knowledge diagram 400 for traffic movement prediction according to a further embodiment.Graph b) of FIG. 3 shows another traffic scenario. Graph b) shows a schematic representation of the same traffic scenario. In the traffic scenario shown, ego vehicle 501 is positioned on a first lane 519. The road user 507 is positioned on a second lane 547. Furthermore, a first further road user 543 and a second further road user 545 are located on a fourth lane 551 of a circular traffic 521.In the traffic scenario shown, the first and the second further road users 543, 545 are located on the same lane 551 and therefore have a longitudinal relative position 553 to one another.The ego vehicle 501 and the road user 507 are located on different lanes which are aligned parallel to one another. The ego vehicle 501 and the road user 507 thus have a transverse relative position 555 with respect to one another.The second lane 547, on which the road user 507 is located, and the fourth lane 551 of the circular traffic 521, on which the first and the second further road user 543, 545 are located, cross. Therefore, the road user 507 and the first and second further road users 543, 545 have a crossing relative position 557.Graph b) schematically shows the relative positions of ego vehicle 501, road user 507 and first and second further road users 543, 545.According to a further specific embodiment of the present invention, the further information about road user 507 or the multiple road users 507, 543, 545, which are situated in the surroundings of ego vehicle 501, may have the aforementioned relative positions with respect to one another and relative to ego vehicle 501.FIG. 4 is a schematic diagram of a knowledge diagram 400 for traffic motion prediction according to an embodiment.The knowledge diagram 400 includes multiple nodes of 401, and edges 403 connecting the nodes of 401.On the right-hand side of FIG. 4, the information about the road users 507 with respective nodes 401 is shown. On the left-hand side of FIG. 4, the information about the movement track 515 of the road user 507 with respective nodes and edges 401, 403 is taken into account.On the right hand side, the road user 507 is shown with a node 401. A plurality of further nodes are connected to the node of the road user 507. The road users 507 can be characterized more accurately via these further nodes.For example, road user 507 may be characterized as a static object 561 or as a movable object 583. Both objects 561, 583 are included in the knowledge diagram 400 via a single node 401. The static object 561 may be, for example, a bicycle stand 563. The movable objects 683 can be, for example, a barrier 583, a slide / dragable object 585, a traffic cone 587, a debris 581. Each of the object types is provided with a respective individual node 401.In addition to the static object 561 or the movable object 683, the road user 507 can also be a vehicle 565. The vehicle 565 may be a bicycle 567, a motorcycle 569, a truck 571, or a car 531. These vehicle types are each provided with respective nodes 401 and connected via respective edges to the node of the vehicle 565.In addition to the vehicle 565, the road user 507 can also be an animal 573 or a human 529. The human 529 may be a policer 575, an adult 577, or a child 579.All nodes 401 on the right side of the knowledge diagram 400 are characterizations of road users 507 and are organized as subclasses 559. Thus, the respective nodes 401 are connected via subclass relations 559 to the respective nodes 401 of the main classes and / or the road user 507.The knowledge diagram 400 is organized into a first scene 533. All nodes of 401 shown are organized with a respective scene 533. For multiple scenes 533, 535, the knowledge diagram 400 may include multiple layers of the nodes 401 shown in FIG. 4. Each plane describes a single scene 533, 535 and is connected to a plane representing a previous and / or subsequent scene 533, 535.This connection of scene 533 to previous and / or subsequent scenes is illustrated in knowledge diagram 400 via a hatVorherige Relationship 599, and to a next scene via a hatNächste Relationship 597.The previous and next scenes relate to the respective time stamp t 1, t 2 to which the respective map data 509 and environment sensor data 503 have been connected.Scene 533 is part of a sequence 537 and is connected to respective node 401 of sequence 537 via a hatErsteSc relationship 601, a hatLastSc relationship 603, and a hatSzene relationship 605.Sequence 537 represents a plurality of successive scenes 533. The sequence 537 is part of a trip 607. Trip 607 includes a plurality of consecutive sequences 537 and is connected to the node of sequence 537 via a hasSeq relationship 611.Trip 607 is connected to node 401 of ego vehicle 501 via a hasEgoVeicle relationship 609.Node 401 of scene 533 is further connected to node of scene participant 589 via a hasSzenenParti relationship 595. The scene participant 589 is also connected to the road participant 507 via an isSzenenParti relationship 591.A scene participant describes a road participant 507 of a specific scene 533, 535. With regard to the respective traffic scenarios, the number of road users 507 is positioned.Scene participant 589 is also linked to itself in a next scene via nextSc relationship 593. Scene participant 589 is also connected to point 613 via a hatPosition relationship 685. This can describe a specific position of the scene participant 589.Point 613 is connected via a subclass relationship 559 to a node 401 of a geometry 635. Point 613 is also connected to a track 519 via an isPointOnLane relationship 657. Trace 519 is connected via a subclass relationship 559 to node 401 of a motion trace 515.Trace 519 is also connected to a next trace via a hasExtLane relationship 617. Trace 519 may further be a right trace or a left trace and thus includes a hatLinkeLane relationship 687, and a hatRightLane relationship 615.Lane 519 may further be a subclass of road crossing element 619 and, therefore, may be connected to respective node 401 of road crossing element 619 via subclass relationship 559.Lane 519 is further connected to a road block 627 via an IsLaneOnRo block relationship 645. The road block 627 is connected to the road element 653 via a subclass relationship 559.Road block 627 is further connected to road segment 623 via is StreetBlockOnRoSegment relationship 629.Road segment 623 is also connected to lane link 621 via isConnectorOnRoSegment relationship 625. Road segment 623 is also connected to intersection 681 via subclass relationship 559.Trace link 621 is further connected to trace 519 via a ConnectAnkommendeLane relationship 647 and a ConnectAbgehendeLane relationship 649.Lane 519 is also connected to a parking space area 631 via a parking space area NextTo relationship 633.The parking space area 631 is further connected to a lane divider via a parking space area hatForm relationship 679.Trace 519 is also connected to polygon 637 via a LaneHasPe relationship 639.Track 519 is also connected to track splitter 525 via a hasRightLaneDi relationship 641 and hasRightTrackTeiler hasLeftLaneDivider 643.The road element 653 also has an isPartOf relationship 651. Moreover, road element 653 is connected to lane segment 655 via subclass relationship 559.Lane section 655 also has a SwitchViaSingleY relationship 659, a SwitchViaRoDi relationship 661, a SwitchViaDoSolidY relationship 663, a SwitchVia relationship 665, a SwitchViaSingleZWhIte relationship 667, a SwitchViaSingleIte relationship 669, a SwitchViaNonVis relationship 671, and a SwitchViaDoHeWh relationship 673, and is connected to a next lane section via a hasExtLaneSn relationship 675.Track portion 655 is further connected to track 519 via a hasLaneSn relationship 677.The embodiment illustrated in FIG. 4 is only one example of a possible knowledge diagram 400 for traffic motion prediction. According to the present invention, knowledge diagram 400 includes at least information about a road user 507 located in the environment of ego vehicle 501 and information about travel lane 515 of this specific road user 507. The respective information enters the knowledge diagram 400 via a plurality of nodes 401 and edges 403.As explained above, the structure of the knowledge diagram 400, in particular the information stored in the nodes 401 and the respective relationships between the nodes 401, is dependent on the respective traffic scenario, the number of road users 507 located in the environment of the ego vehicle 501 and the information accessible via the environment sensor data 503 and the map data 509.In the present invention, the concepts sequence 537 and scene 533, 535 are used. A detailed description of the concepts sequence 537 and scene 533, 535 can be found in Lavdim Halilaj, Jurgen Luettin, Cory Andrew Henson, Sebastian Monka, Knowledge Graphs for Automated Driving, 2022 IEEE Fifth International Conference on Artificial and Knowledge Engineering (AIKE), pages 98-105, 2022.In short, a scene 533, 535 relates to a single moment in a traffic situation. The relationship hasTimeAmp is an integer data property with a unix time stamp defining a particular time.A sequence 537 is an ordered collection of scenes 533, 535. A sequence 537 can be thought of as a video whose frames are scenes 533, 535. Since the order is inherent to scenes 533, 535, object properties hasExtScreen 597 and hasPreScreen 599 are defined to associate successive scenes 533, 535. The sequence 537 and scene 533, 535 instances may be generated from the map data 509 of the electronic road map 511. The map 511 may be based on the record of nuSces.Equations (1) through (22) are formulated in the formalism of the description logic. A detailed explanation of formalism can be found in Krötzsch et al. arXiv:1201.4089.Equation (1) provides a definition of scene 533, 535 and sequence 537. Scene 533, 535 and sequence 537 are both nodes 401 in knowledge diagram 400.Sequences 537 relate to certain motion prediction situations. During the recording of movement data, the ego vehicle may be driving for hours and record 537 multiple sequences.A trip 607 is such a capture session and each of its entities points to multiple sequences 537. Each trip 607 is undertaken in a particular region of interest, a location associated therewith over hatOrt. hasRightHandleRaff is a boolean property for describing the direction of travel at a location.Equations (3), (4) provide a definition of the concepts trip 607 and location.Trip 607 instances can be generated from map data 509, such as the nuSc LOG records, and a location instance can be manually created for each of the four maps.The road user 507 concept represents a traffic agent that is present in one or more scenes 533, 535. The different types of subscribers are modeled as subclasses of the subscriber concept. Overall, there are 23 different ones.Examples are cars, adults, children, people, hospital cars, bicycles, and so on. Participants may refer to an entity at a particular time. NextScene has introduced a new relationship to link entities over a period of time.Moreover, the concept SceneParti was introduced as a presentation of an agent at a certain time, and the meaning of the road user 507 was changed to present an agent generally and independently of time.This prevents time-independent information, e.g. the size of agents, from having to be stored redundantly. The semantic relationship between ScenePartis may be modeled such that agents may follow each other, possibly overlap, or be parallel to each other.Equations (5), (6) provide a definition of the concepts SceneParti and Partial.SceneTeilnehmer instances are generated from SAMPLE_ANNOTATION and EGO_POSE records. The EGO_POSE data sets are needed to enable the ego vehicle to be recorded as a SceneParti. This is a novel in the provided data representation. Previous work has ignored the effect of the ego vehicle 501 on the motion of the target vehicle.The currently provided data analysis of the nuSc dataset shows that the ego vehicle 501 and the destination may be close in a considerable number of cases up to 2 m. Therefore, it is considered that the subject vehicle 501 has a large influence on the behavior of the target vehicle.The central component of the road traffic infrastructure is the lane 519, 547, 549, 551. This is a non-overlapping section of the road surface, typically bounded by road boundaries, on which only one direction of travel is permitted. This is a natural, physical lane configuration, as one would not consider intersections or merging points to be part of a particular lane 519, 547, 549, 551.Another modeling option would be to use a definition that handles lanes 519, 547, 549, 551 as not physical, but logical units that may overlap. Such tracks 519, 547, 549, 551 would pass over intersections. For reasons of compatibility with nuSc, we have chosen the natural, physical definition. To maintain the logical connectivity information with the natural definition, one needs lane connectors that have the functional properties hasComLane and hasOutLane, pointing to a lane, respectively.Equations (7), (8) provide a definition of the concepts Lane and LaneConnect.Lane and LaneConnect instances are generated from LANE and LANE CONNECTOR records, respectively. LaneSn, switch. Lane boundaries are another critical element that determines the travel of cars.There are different kinds of lane separators, for example, solid lines and dotted lines. A track section or LaneSnpnet is defined as part of a track 519, 547, 549, 551, which has a single boundary type on each of its left and right sides. This allows for the introduction of a switch via property for each frame type, i.e. switchViaDoubleDashed switchViaSingleSolid, etc. Adjacent sections, for example having a single fixed boundary between them, are related via switchViaSingleSolid, which represents that a continuous boundary would have to be crossed in order to change from one lane to another.Changes over boundaries that are illegal are maintained in the model because cars may sometimes violate traffic rules and, for example, pass over a continuous boundary, for example, hasNextLaneSnippet points from one patch to the next immediately following, keeping them ordered, and hasLaneSnpette holds them connected to their parent lane 519, 547, 549, 551.Further, since experimental findings have shown that it is important for the performance of trajectory prediction to keep the sections short, they are further divided if they are longer than 20 meters. SectionHatLength, or snippetHasLength records how long a particular track section is.Equation (9) provides a definition of the concept LaneSnppet.LaneSn instances were computed from LANE records. The boundary types (solid line, dashed line, etc.) on each side of a track have been tracked and split into sections that have invariable boundary types on both sides. Sections were split as needed to meet the 20 m length limit. Thereby, LaneSn instances with constant boundary types were created on both sides. switchVia edges were placed between adjacent LaneSn instances.To represent the centerlines (where cars normally travel) of lanes and lane connections, a sequence of poses is used. A pose consists of a position and an orientation. Orientation here means the orientation of the lane, i.e. the direction of traffic, at a specific position. An OrderedPose is a subclass of Pose that also has the textithasExtPose property.This is used to order them and define the typical trajectory along a LaneConnect via the ConnectorHasPos relationship to all their ordered poses. The position of a pose is modeled with sf:point, as are the agent positions, and its orientation with the data property poseHasOr, is represented as an angle between the positive x-axis and the orientation (yaw). In contrast to lane connections, the lane model must also meet broad competence issues.The natural name LaneDisk is chosen to represent the combination of center position and lane width. hasExtLaneDisk provides the order by connecting successive disks, hasLaneDisk points from the parent lane to its disks, and LaneDiskHatHasWidth is the data property that the name suggests.Equations (10), (11), (12) provide a definition of the concepts LaneSlice and OrderedPose.OrderedPose instances were generated from the manually annotated arc lines of nuSc with a resolution of 2 m using the Nu Devkit. LaneSlice instances additionally represent the lane width.A given center point whose width is to be calculated is projected on both the left and right boundaries. The projected points are those points on the boundaries that have the smallest Euclidean distance to the given center point. The width is obtained from the distance between the projected points. StopAr.Stop areas are a very important concept for motion prediction because by definition they are the areas where cars tend to come to a stop. There are several reasons for such regions, and each is modeled as a subclass of the parent class StopAr. These include stop signs, forward travel signs, oncoming traffic when turning left, pedestrian crossings and traffic lights. CausesStopBeiCausesStopAt, the originating entity associates with the associated StopAr.Equation (13) provides a definition of the concept StopAr.StopAr instances were generated from nuSc STOP LINE records. TrafficLight. Its physical appearance is represented by the categorial data property hasTrafficLightType that distinguishes between horizontally and vertically stacked traffic lights. In addition, the traffic lights are at a particular position and are aligned in a particular direction, which is illustrated by trafficLightHasPose pointing to a particular pose instance. The dynamic state of traffic lights (light color) is not modeled because this information is not available in the nuSc record.Equation (14) provides a definition of the concept TrafficLight.TrafficLight instances were generated from nuSc TRAFFIC LIGHT records.Pedestrian Crossing. Here, pedestrians can legally cross the road. The two sidewalks connected via an intersection are represented with the connectGeh relationship.Equation (15) provides a definition of the concept PedCross.The inspection of several pedestrian crossings and sidewalks in the nuSc dataset revealed that they do not necessarily touch each other, but are always in the vicinity.Heuristics were considered to be sidewalks around 5 m around an intersection. Our algorithm selects the two sidewalks with minimum distances. In order to check the correctness of the implementation, a subset of the triples generated was visualized and checked manually. Sidewalk, parking space area.Sidewalks are modeled with an equally namily concept. A car parkAr is any area in which cars can park, whether in an actual parking lot or on the roadside. In order to explicitly represent the proximity between adjacent road parts, there is isExtTo. carparkArealExtTorepresents the lane next to which a parking space is located, and accordingly walkwaylsExtTorepresents the proximity between sidewalks and lanes.Equations (16), (17) provide a definition of the concepts WalkWay and CarparkAr.The isExtTo relationship between sidewalks, lanes, and parking spaces is generated for the entity pairs that are within 4 m.This heuristic threshold was chosen after visualizing multiple lanes, parks, and sidewalks and their vicinity. In this way, an explicit spatial relationship is established between adjacent roadway surfaces. Street blocks 627 group adjacent lanes that are in the same direction. A hasExtRoBlock edge exists from one block to another if they contain successive tracks.Street block 627 connectivity therefore models any potential future region that a car can travel in consideration of possible lane changes, which is not the case with lane connectivity. Moreover, a hasOpRoBlock relationship is introduced. It exists between two road blocks 627 when they are parallel to each other on the same road and carry traffic in opposite directions. This extends the spatial connectivity in the diagram, such as the isExtTo property, and makes spatial relationships clear that people can intuitively recognize.Equation (18) provides a definition of the concept RoadBlock.RoadBlock instances were calculated by grouping adjacent lanes, and connectivity between street blocks 627 was determined by lane connectivity. Instances could not be generated from nuSc ROAD BLOCK 627 records because they contained faulted shapes on two of the four cards as addressed in a GitHub problem. Intersection. Here, a plurality of tracks cross each other.The typical paths that pass over intersections are characterized by transverse track connections. A lane 519, 547, 549, 551 leading into the intersection with a lane 519, 547, 549, 551 leading out of the intersection just when a car is allowed to drive from the former to the latter across the intersection. isConnectOnRoSegment links intersections with the lane links thereon.Equation (19) provides a definition of the concept intersection.Intersection instances were generated from ROAD SEGMENT records. The explicit spatial connection between them and lane connections was calculated by checking whether a lane connection overlaps an intersection. hasForm. To model the exact positions, shapes and sizes of all map elements described above, has been introduced for each individual hasSha relationship.Each form is represented by a subclass of the GeoSPARQL Simple Features (prefix sf) 'sf:Geometry ontology concept. This includes, for example, sf:polygon, with which polygonal structures such as sidewalks, lanes or intersections can be modeled. Data characteristics of geometries store their exact shapes in nuSc coordinates (x, y), but also in GPS coordinates representing the actual location on the earth.This allows fusion with other geographic data sources and geodata analyses. isOn, AreaElement. To establish a connection between agent and card, the is On relationship is introduced. AreaElement is defined as a superclass for all map elements that occupy an area, i.e., have an sf:polygon geometry. isOn links a scene participant to the map object on which it is currently located.Equation (20) provides a definition of the concept AreaElement.A graphical representation of the nSKG is created as an easily loadable data set. It is in the format of PyTor Geometric (PyG) diagrams. PyTor Geometric (PyG) is one of the most widely used libraries for graphical neural networks. Therefore, the data set was created in a PyG compatible format that can be easily loaded by PyG data loaders. Formally, a heterogeneous digram G=(V, E, τ, φ) has nodes v ∈V with node types τ(v), and edges (u, v) ∈E with edge types φ(u, v). The edges are directed because they are based on properties of the knowledge diagram.Each example i in the constructed dataset is a pair (x i, y i) ∈(G, R 12), where x i is a scene diagram with trajectory information of the last two seconds, a local map and a destination identifier, and y i is the future ground truth trajectory of the destination. This makes our data set a diagram regression task.The constraints of 2 seconds in the past and 6 seconds in the future (sampled at 2 Hz) are maintained by nu, so that all results of our new graph record can be compared to those of the nuSc raw data. The training, validation and test distributions of nuSc are also preserved.The coordinate system used was an important consideration because proper selection of the coordinate system allows the passage of a data-centered inductive trend, namely displacement and rotational invariance. Inductive distortions are generally considered essential for a good generalization of deep learning.Coordinates in the knowledge diagram (and in nuSc) are initially located in a global coordinate system. These were separately transformed into local scene diagram specific coordinates for each scene diagram, with the origin at the location of the target agent and the positive x-axis along the direction of sight of the target. More specifically, let p Ziel and R Ziel be the global position (vector) and orientation (rotation matrix), respectively, of the target vehicle in the scene diagram g. let p global, R global be any global position and global orientation, respectively. Its representation in the local frame is given byEquations (21), (22) provide a definition of concepts P lokal and R lokal. where R -1target is the inverse of the rotation matrix R target. In this way, the coordinates of all entities in g can be transformed into the local coordinate system. Predictions will automatically be shift and rotation invariant because all shifts and rotations are removed during the transformation.In all examples, the target is at the origin and oriented along the positive x-axis. It has been shown empirically that this transformation improves the performance of trajectory prediction.The trajectory information in a scene diagram includes the nodes a sequence 537, scene, scene participants, and participants, as well as the semantic relationships between scene participants. The object properties between them in the knowledge diagram become heterogeneous edges.Their data properties are converted to node features. SPARQL queries were used to retrieve the last two seconds of scene instances and the relevant agents therein. Relevant agents are those that can influence the movement of the target vehicle defined as those that are on a part of the relevant extracted map described below. This does not take account of scene participants already behind the target agent, which is an improvement over previous works.In addition to trajectory information, a scene diagram also contains the abundance of map information modeled in our ontology. However, the inclusion of whole city plans is counterproductive and would make the graphics unnecessarily large. The larger a diagram, the more extensive dependencies may occur, which poses problems for modern graphical neural networks. Only those parts of the map representing potential paths of the destination have been considered. To extract it from the knowledge diagram, a destination is assigned to the street block 627 on which it is located and the hasExtRoBlock edges are tracked four blocks into the future.As relevant length, four street blocks 627 were selected after analyzing the distances traveled by agents in nuSc within 6 seconds. Road blocks 627 typically have the length of a lane segment or more, and a lane connection is about 5-10 m long, so four road blocks 627 in most cases cover the future trajectory, as agents normally travel no more than 120 m, as our analysis yielded. If one were to add more in the future than necessary, the plots would become larger than necessary, which would impair the performance of current graphic neural networks. The surrounding map units of the potential paths, sidewalks, parking spaces, pedestrian crossings and stop areas, etc., are extracted via the explicit spatial relationships described in the previous section that were modeled in ontology.These explicit spatial relationships are also retained in the heterogeneous scene diagrams and are transformed into heterogeneous edges as described above, like all other object properties. The connection between the extracted trajectory information and the map is made via the isOn relation as in the knowledge diagram.The created chart record contains more than 40,000 scene charts and ground truth futuresheets. Each scene diagram contains an average of between 1,000 and 2,000 nodes. Graphical neural networks trained on these scene diagrams will have access to more information than any prior trajectory prediction method. Current prior art approaches are likely to be exceeded with this representation by a suitable graphical neural network, since competing approaches do not have access to all information important for trajectory prediction.FIG. 5 shows a flow diagram of a method 100 for generating a knowledge diagram 400 for traffic motion prediction according to an embodiment.In order to generate a knowledge diagram 400 for traffic movement prediction, environment sensor data 503 are received from at least one environment sensor 505 of the ego vehicle 501 in a first method step 101. Environment sensor data 503 represents the environment of ego vehicle 501 and includes information about at least one road user 507 located in the environment of ego vehicle 501.Environment sensor 505 may be, for example, a video camera, a radar sensor, an ultrasonic sensor, or any other conventional environment sensor used in automated driving.In a second method step 103, map data 509 of an electronic road map 511 are received. Map data 509 represent a road network 513 in the environment of ego vehicle 501 and include information about at least one movement lane 515 on which road user 507 is located.The road user 507 may be a static object, a moving object, a human, an animal, a vehicle, a car, a truck, a roadway, a motorcycle, a bicycle, a barrier, a traffic cone or any other object that can be detected in a normal traffic scenario via the above-mentioned environment sensor 505.Depending on the type of road user 507, the travel lane 515 may be a road 517, a lane 519, 547, 549, 551, an intersection, an underpass, a bridge, a freeway, a freeway entrance, a freeway exit, a circumcircle 521, a parking lot, a parking area, a bicycle path, a highway track, a pedestrian crossing 523, a sidewalk, or any other infrastructure in the environment of the ego vehicle 501 that enables movement of the road user 507.In a further method step 105, the information about the at least one road user 507 and the information about the movement track 515 are extracted from the environment sensor data 503 and the map data 509.In a further method step 109, a time stamp t 1, t 2 of environment sensor data 503, 509 is determined.In a further method step 111, environment sensor data 503 and map data 509 are organized into scenes 533, 535 and sequences 537. A scene 533, 535 comprises the information of environment sensor data 503 and map data 509 at a time stamp t 1, t 2. Sequence 537 is a series of successive scenes 533.In a further method step 113, nodes 401 of knowledge diagram 400 are organized with regard to scenes 533, 535 and sequences 537 of surroundings sensor data 503 and map data 509.In a further method step 107, the knowledge diagram 400, which comprises a plurality of nodes 401 and edges 403, is generated based on the extracted information relating to the road user 507 and the movement track 515 of the map data 509 and the environment sensor data 503. The knowledge diagram 400 comprises at least nodes 401 which represent the information regarding the road user 507 and nodes 401 which represent information regarding the movement track 515.According to one embodiment, further information regarding the motion track 515 is included in the knowledge diagram 400.The further information may include road geometries, lane geometries, lane dividers, lane boundaries, lane connections, intersections, stopping areas, traffic signals, traffic signs, traffic rules, road conditions, slope values, pedestrian crossings, parking lot areas, road sections, road blocks 627, or any other feature of possible motion lanes 515 that may occur in a normal traffic scenario and may affect the motion of the respective road users 507.In addition to the features of the road user 507 mentioned above, further features with regard to a relative position of the road user 507 either with respect to further road users or with respect to the ego vehicle 501 can be recorded. The relative positions may include a relative longitudinal position 553, a relative transverse position 555, and / or a relative cut position 557.According to a further embodiment, the information of the map data 509 taken into account in the knowledge diagram 400 is limited to a region of a possible path of the road user 507. Information about areas in the road network 513 that are not reachable by the road user 507 from his current position is not included in the knowledge diagram 400. This reduces the complexity of the knowledge diagram 400 and simplifies the generation of the knowledge diagram 400 as well as the computational power and storage capacity 400 required for the generation and operation of the knowledge diagram.According to one embodiment, the method 100 for generating a knowledge diagram 400 for traffic movement prediction is carried out during an operating mode, in particular during a driving mode, of the ego vehicle 501.FIG. 6 shows a flow diagram of a method 200 for traffic motion prediction according to an embodiment.In order to carry out a motion prediction, a knowledge diagram 400 for traffic motion prediction is generated in a first method step 102 by carrying out the method 100 for generating a knowledge diagram 400 for traffic motion prediction according to the above-mentioned embodiments.In a further method step 203, a future movement of at least one road user 507 positioned in a surrounding area of the ego vehicle 501 is predicted by a movement prediction module 539 on the basis of the knowledge diagram 400.According to an embodiment, the motion prediction module 539 includes a trained artificial intelligence capable of predicting the motion of the road user 507 based on the information of the knowledge diagram 400. The trained artificial intelligence may comprise a trained diagram network.FIG. 7 shows a flow diagram of a method 300 for controlling an ego vehicle 501 according to an embodiment.For controlling an ego vehicle 501, a future movement of at least one road user 507 located in a surrounding area of the ego vehicle 501 is predicted in a first method step 301 by carrying out the method 204 for traffic movement prediction according to the above-mentioned embodiments.In a further method step 303, at least one control function of the ego vehicle 501 is carried out on the basis of the predicted movement of the at least one road user 507.The control function of the ego vehicle 501 may include a steering function and / or acceleration or deceleration function and / or deceleration function of the ego vehicle 501.FIG. 8 shows a schematic representation of a computer program product 700 having instructions which, when executed by a data processing unit, cause the program to execute the method 100 for generating a knowledge diagram for traffic motion prediction and / or the method 200 for traffic motion prediction and / or the method for controlling an ego vehicle 501.In the embodiment shown, the computer program product 700 is stored on a computing unit or a storage medium 701. Storage medium 701 may be any storage medium known in the art.References included in the specificationThis list of documents cited by the applicant has been produced in an automated manner and is only included for the better information of the reader. The list is not part of the German patent application or utility model application. The DPMA does not take any adhesion for any faults or omissions.Cited Non-Patent LiteratureLavdim Halilaj, Jurgen Luettin, Cory Andrew Henson, Sebastian Monka, Knowledge Graphs for Automated Driving, 2022 IEEE Fifth International Conference on Artificial and Knowledge Engineering (AIKE), pages 98-105, 2022

[0111] Kretzsch et al. arXiv:1201.4089

[0114]

Claims

Computer-implemented method (100) for generating a knowledge diagram (400) for traffic movement prediction, comprising: receiving (101) environment sensor data (503) of at least one environment sensor (505) of an ego vehicle (501), wherein the environment sensor data (503) represent the environment of the ego vehicle (501) and comprise information relating to at least one road user (507) located in the environment of the ego vehicle (501); receiving (103) map data (509) from an electronic road map (511), wherein the map data (509) represent a road network (513) in the environment of the ego vehicle (501) and comprise information relating to at least one movement lane (515) on which the road user (507) is located; Extraction (105) of the information about the at least one road user (507) from the environment sensor data (503) and extraction of the information about the movement lane (515) on which the road user (507) is located from the map data (509); and generation (107) of a knowledge diagram (400) of the road network (513) in the environment of the ego vehicle (501), comprising nodes (401) and edges (403), based on the map data (509) and / or the environment sensor data (503), wherein the knowledge diagram (400) comprises at least one node (401) which represents the road user (507) and at least one node (401) which represents the lane on which the road user (507) is located.The method (100) of claim 1, wherein the travel lane (515) on which the road user (507) is located is at least one of the following list: a road (517) and / or lane (519, 547, 549, 551), an intersection, an underpass, a bridge, a freeway, a freeway entrance, a freeway exit, a circular traffic (521), a parking bay, a parking lot, a bicycle path, a road lane, a pedestrian crossing (523), a sidewalk.Method (100) according to Claim 1 or 2, wherein the map data (509) of the electronic road map (511) comprise further information on further features of the movement track (515) on which the road user (507) is located, and wherein the at least one further feature of the movement track (515) of the road user (507) is integrated into the knowledge diagram (400) via at least one further node (401).Method (100) according to Claim 1 or 2, wherein the environment sensor data (503) further comprise further information about at least one further feature of the road user (507), and wherein the at least one further feature of the road user (507) is integrated into the knowledge diagram (400) via at least one further node (407).Method (100) according to claim 3 or 4, wherein the nodes (401) and further nodes (405, 407) of the knowledge diagram (400) are organized into classes and subclasses.Method (100) according to one of Claims 3 to 5, wherein the further features of the movement track (515) are at least one from the following list, comprising: road geometries, track geometries, track dividers (525), track boundaries (527), road connections, intersections, holding regions, traffic signals, traffic signs, traffic rules, road conditions, gradient values, pedestrian crossings, parking space regions, road sections, road blocks (627).Method (100) according to one of Claims 4 to 6, wherein the further features of the road user (507) are at least one from the following list, comprising: a static object, a moving object, a human (529), an animal, a vehicle, a passenger car (531), a truck, a roadway, a motorcycle, a bicycle, a barrier, a traffic cone, a relative position (547, 549, 551) with respect to at least one further road user (543) and / or with respect to the ego vehicle (501).The method (100) according to any one of the preceding claims, further comprising: determining (109) a time stamp (t1, t2) of the environment sensor data (503) and the map data (509); organizing (111) the environment sensor data (503) and the map data (509) into scenes (533, 535) and sequences (537), wherein a scene (533, 535) comprises the information of the environment sensor data (503) and the map data (509) for a time stamp (t1, t2), and wherein a sequence (537) is a series of consecutive scenes (533); and organizing (113) the nodes (401) of the knowledge diagram (400) with respect to the scenes (533, 535) and sequences (537) of the environment sensor data (503) and the map data (509).Method (100) according to one of the preceding claims, wherein the information of the map data (509) taken into account in the generation of the knowledge diagram (400) is restricted to a region of a possible path of the road user (507).Method (100) according to one of the preceding claims, wherein the method (100) is carried out during a driving mode of the ego vehicle (501).A computer-implemented method (200) for traffic movement prediction, comprising: generating (201) a knowledge diagram (400) for traffic movement prediction by executing the method (100) for generating a knowledge diagram (400) for traffic movement prediction according to any one of the preceding claims 1 to 10; predicting (203), by a movement prediction module (539), a future movement of at least one road user (507) located in a vicinity of the ego vehicle (501) based on the knowledge diagram (400).The method (200) of claim 11, wherein the motion prediction module (539) comprises a trained artificial intelligence capable of predicting the motion of the road user (507) based on information of the knowledge diagram (400).A computer-implemented method (300) for controlling an ego vehicle (501), comprising: predicting (301) a future movement of at least one road user (507) located in a surrounding area of the ego vehicle (501) by executing the method (200) for traffic movement prediction according to any one of the preceding claims 11, 12; and executing (303) at least one control function of the ego vehicle (501) based on the predicted movement of the at least one road user (507).Computing unit (541) equipped for carrying out the method (100) for generating a knowledge diagram (400) for traffic movement prediction according to one of Claims 1 to 10 and / or the method (200) for traffic movement prediction according to one of the preceding Claims 11, 12 and / or the method for controlling an ego vehicle (501) according to Claim 13.A computer program product (700) comprising instructions which, when the program is executed by a data processing unit, cause the data processing unit to execute the method (100) for generating a knowledge diagram (400) for traffic motion prediction according to any one of the preceding claims 1 to 10 and / or the method (200) for traffic motion prediction according to any one of the preceding claims 11, 12 and / or the method for controlling an ego vehicle (501) according to claim 13.

Images