Method and system for predicting vehicle movement trajectory

Abstract

A method and system for predicting a moving trajectory of a vehicle are provided. The obstacle state evolution of the spatial position of a moving obstacle over a period of time is determined. The obstacle relationship evolution between the moving obstacle and each of one or more lanes near the moving obstacle over a period of time is also determined. The expected movement of the moving obstacle is predicted based on the obstacle state evolution and the lane-obstacle evolution. Thereafter, based on the predicted expected movement of the moving obstacle, the trajectory of the ADV is planned to control the ADV to avoid collision with the moving obstacle. The above process is iteratively performed for each of the moving obstacles detected within a predetermined proximity of the ADV.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to operating an autonomous vehicle. More specifically, embodiments of the present disclosure relate to predicting the movement or trajectory of a moving obstacle. Background Art

[0002] A vehicle operating in an autonomous mode (e.g., without human control) can relieve occupants (particularly the driver) from certain driving-related responsibilities. When operating in an autonomous mode, the vehicle can navigate to various locations using onboard sensors, allowing the vehicle to travel with minimal human interaction, or in some cases without any passengers.

[0003] Motion planning and control are crucial operations in autonomous driving. When planning the path or trajectory of an autonomous driving vehicle (ADV), it is important to predict the movement or trajectory of each moving obstacle within the vicinity of the ADV. However, such predictions are difficult to achieve with high accuracy. For example, the trajectory of a moving obstacle can be determined by the intention of the driver of the moving obstacle, which is affected by the surrounding static environment (such as the scenery, lane lines, and road shapes near the moving obstacle). Therefore, it may be difficult to estimate the intention of the driver of the moving obstacle. Summary of the invention

[0004] In a first aspect, a computer-implemented method for operating an autonomous vehicle (ADV) is provided, the method comprising:

[0005] Determine the spatial position of the moving obstacle and the evolution of the obstacle state over a period of time;

[0006] determining an evolution of a lane-obstacle relationship between a moving obstacle and each of one or more lanes in the vicinity of the moving obstacle over a period of time;

[0007] Predicting the expected movement of a moving obstacle based on the evolution of the obstacle state and the evolution of the lane-obstacle relationship; and

[0008] Based on the predicted expected movement of the moving obstacle, a trajectory of the ADV is planned to control the ADV to avoid collision with the moving obstacle.

[0009] In a second aspect, a non-transitory machine-readable medium is provided having instructions stored therein, which when executed by a processor causes the processor to perform the operations of the method of operating an autonomous vehicle (ADV) as described in the first aspect.

[0010] In a third aspect, there is provided a data processing system comprising:

[0011] Processor; and

[0012] A memory is coupled to the processor and is configured to store instructions that, when executed by the processor, cause the processor to perform the operations of the method of operating an autonomous vehicle (ADV) as described in the first aspect.

[0013] The method and system of the present invention can accurately predict the moving trajectory of a moving obstacle. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Embodiments of the present disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings in which like references indicate similar elements.

[0015] Figure 1 is a block diagram illustrating a networking system according to one embodiment.

[0016] Figure 2 is a block diagram illustrating an example of an autonomous vehicle according to one embodiment.

[0017] Figure 3A-3B is a block diagram illustrating an example of a perception and planning system for use with an autonomous vehicle, according to one embodiment.

[0018] Figure 4 is a block diagram illustrating an example of a prediction module according to one embodiment.

[0019] Figure 5 is a process flow diagram illustrating an example of a prediction module according to one embodiment.

[0020] Figure 6 An example of the problem of predicting the trajectory of a moving obstacle near an exit is shown.

[0021] Fig. 7A An example of predicting pedestrian trajectories based on dynamic interactions via a grid-like Euclidean method is shown.

[0022] Figure 7B An example of predicting pedestrian trajectories based on dynamic interactions via a non-Euclidean approach is shown.

[0023] Figure 7C An example of predicting the trajectory of a moving obstacle based on a static environment by a grid-like Euclidean method is shown.

[0024] Fig.7D An example of predicting a trajectory of a moving obstacle based on a static environment according to one embodiment is shown.

[0025] Figure 8 An example of a method of predicting a trajectory of a moving obstacle according to one embodiment is shown.

[0026] Figures 9A-9EDetails of an example of a method of predicting a trajectory of a moving obstacle are shown. Fig.9A indicates receipt of new information; Fig. 9B Showing the time evolution; Fig. 9C Shows spatial aggregation; Fig.9D Shows lane attention; and Fig.9E Shows an update of the overall state of the moving obstacle.

[0027] Fig.10 is a flow chart illustrating a process of predicting a trajectory of a moving obstacle according to one embodiment. DETAILED DESCRIPTION

[0028] Various embodiments and aspects of the present disclosure will be described with reference to the details discussed below, and the accompanying drawings will illustrate various embodiments. The following description and the accompanying drawings are illustrations of the present disclosure and should not be construed as limiting the present disclosure. Many specific details are described to provide a comprehensive understanding of the various embodiments of the present disclosure. However, in some cases, in order to provide a brief discussion of the embodiments of the present disclosure, known or conventional details are not described.

[0029] References in the specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in conjunction with the embodiment can be included in at least one embodiment of the present disclosure. The phrase "in one embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0030] According to some embodiments, by utilizing an attention mechanism and a long short-term memory (LSTM) network, the relationship between the driver's intention and the change of position of a mobile obstacle relative to the road infrastructure is learned. The relationship is used to guide the prediction of the expected movement of the mobile obstacle. For example, the lanes on the highway can be regarded as non-Euclidean structures, and the movement history of the mobile obstacle can be unfolded to form a spatiotemporal graph. The expected movement of the mobile obstacle can be predicted based on a graph neural network. It is advantageous to process the static environmental features around the predicted object using a non-Euclidean method. The practicality and interpretability analysis of the method disclosed herein shows the great potential for large-scale deployment in various autonomous driving systems.

[0031] According to one embodiment, a computer-implemented method for operating an ADV is disclosed. An obstacle state evolution of a spatial position of a moving obstacle over a period of time is determined. A lane-obstacle relationship evolution of the moving obstacle and each of one or more lanes in the vicinity of the moving obstacle over a period of time is also determined. An expected movement of the moving obstacle is predicted based on the obstacle state evolution and the lane-obstacle relationship evolution. Thereafter, based on the predicted expected movement of the moving obstacle, a trajectory of the ADV is planned to control the ADV to avoid collision with the moving obstacle. The above process is iteratively performed for each of the moving obstacles detected within a predetermined proximity of the ADV.

[0032] In one embodiment, a spatiotemporal graph may be generated based on the spatial position of a mobile obstacle and the evolution of a lane-obstacle relationship between the mobile obstacle and each of one or more lanes. The spatiotemporal graph may include two types of nodes, a set of spatial edges, and two types of temporal edges. The two types of nodes may include an obstacle node representing the spatial position of a mobile obstacle at a given time, and a corresponding lane node representing each of a plurality of lanes at a given time. For example, each spatial edge in the set of spatial edges may indicate a corresponding pairwise relationship between an obstacle node and a corresponding lane node. For example, the two types of temporal edges may include the evolution of an obstacle state of a spatial position of a mobile obstacle over a period of time, and the evolution of a lane-obstacle relationship between a mobile obstacle and each of one or more lanes over a period of time. .

[0033] In one embodiment, the obstacle state evolution of the spatial position of the moving obstacle over a period of time is determined by using an LSTM network including a multi-layer perception (MLP) network. In one embodiment, the lane-obstacle relationship evolution of the moving obstacle for each lane in one or more lanes over a period of time is determined by using an LSTM network including an MLP network.

[0034] In one embodiment, an aggregated lane coding indicating an expected movement of a moving obstacle is determined, wherein the expected movement of the moving obstacle is also predicted based on the aggregated lane coding. In one embodiment, for each of the one or more lanes, the lane coding is determined based on an evolution of a lane-obstacle relationship of the moving obstacle to the lane, a current relative position of the lane to the moving obstacle, and a future shape of the lane. In one embodiment, for each of the one or more lanes, an attention score of the lane is determined based on an evolution of a lane-obstacle relationship of the moving obstacle to the lane, and a current relative position of the lane to the moving obstacle. In one embodiment, the aggregated lane coding is based on a weighted sum of lane codings for each of the one or more lanes, wherein the weight of each of the one or more lanes is based on the attention score of the lane.

[0035] Figure 1 is a block diagram showing an autonomous vehicle network configuration according to one embodiment of the present disclosure. Figure 1 , network configuration 100 includes an autonomous vehicle 101 that can be communicatively coupled to one or more servers 103-104 via a network 102. Although one autonomous vehicle is shown, multiple autonomous vehicles can be coupled to each other and / or to servers 103-104 via network 102. Network 102 can be any type of network, such as a local area network (LAN), a wide area network (WAN) (such as the Internet, a cellular network, a satellite network, or a combination thereof), a wired or wireless network. Servers 103-104 can be any kind of server or server cluster, such as a web or cloud server, an application server, a back-end server, or a combination thereof. Servers 103-104 can be data analysis servers, content servers, traffic information servers, maps and points of interest (MPOI) servers, or location servers, etc.

[0036] An autonomous vehicle refers to a vehicle that can be configured to be in an autonomous mode, in which the vehicle navigates through an environment with little or no input from a driver. Such an autonomous vehicle may include a sensor system having one or more sensors configured to detect information about the environment in which the vehicle is operating. The vehicle and its associated controller(s) use the detected information to navigate through the environment. Autonomous vehicle 101 may operate in a manual mode, a fully autonomous mode, or a partially autonomous mode.

[0037] In one embodiment, autonomous vehicle 101 includes, but is not limited to, perception and planning system 110, vehicle control system 111, wireless communication system 112, user interface system 113, and sensor system 115. Autonomous vehicle 101 may also include certain common components included in ordinary vehicles, such as an engine, wheels, a steering wheel, a transmission, etc., which may be controlled by vehicle control system 111 and / or perception and planning system 110 using various communication signals and / or commands, such as, for example, acceleration signals or commands, deceleration signals or commands, steering signals or commands, braking signals or commands, etc.

[0038] The components 110-115 may be communicatively coupled to each other via an interconnect, a bus, a network, or a combination thereof. For example, the components 110-115 may be communicatively coupled to each other via a controller area network (CAN) bus. The CAN bus is a vehicle bus standard designed to allow microcontrollers and devices to communicate with each other in applications without a host computer. It is a message-based protocol originally designed for multi-way electrical wiring within automobiles, but is also used in many other situations.

[0039] Reference now Figure 2 In one embodiment, the sensor system 115 includes, but is not limited to, one or more cameras 211, a global positioning system (GPS) unit 212, an inertial measurement unit (IMU) 213, a radar unit 214, and a light detection and ranging (LIDAR) unit 215. The GPS system 212 may include a transceiver 214 that is operable to provide information about the location of the autonomous vehicle. The IMU unit 213 may sense changes in the position and direction of the autonomous vehicle based on inertial acceleration. The radar unit 214 may represent a system that uses radio signals to sense objects within the local environment of the autonomous vehicle. In some embodiments, in addition to sensing objects, the radar unit 214 may additionally sense the speed and / or heading of the object. The LIDAR unit 215 may use lasers to sense objects in the environment in which the autonomous vehicle is located. The LIDAR unit 215 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. The camera 211 may include one or more devices to capture images of the environment surrounding the autonomous vehicle. The camera 211 may be a still camera and / or a video camera. The camera may be mechanically movable, for example by mounting the camera on a rotating and / or tilting platform.

[0040] The sensor system 115 may also include other sensors, such as sonar sensors, infrared sensors, steering sensors, throttle sensors, brake sensors, and audio sensors (e.g., microphones). The audio sensor may be configured to capture sound from the environment surrounding the autonomous vehicle. The steering sensor may be configured to sense the steering angle of the steering wheel, the vehicle wheels, or a combination thereof. The throttle sensor and the brake sensor sense the throttle position and the brake position of the vehicle, respectively. In some cases, the throttle sensor and the brake sensor may be integrated into an integrated throttle / brake sensor.

[0041] In one embodiment, the vehicle control system 111 includes, but is not limited to, a steering unit 201, a throttle unit 202 (also referred to as an acceleration unit), and a brake unit 203. The steering unit 201 is used to adjust the direction or forward direction of the vehicle. The throttle unit 202 is used to control the speed of the motor or engine, thereby controlling the speed and acceleration of the vehicle. The brake unit 203 is used to decelerate the vehicle by providing friction to slow down the wheels or tires of the vehicle. Note that Figure 2 The components shown in may be implemented in hardware, software, or a combination thereof.

[0042] Reference again Figure 1, the wireless communication system 112 is used to allow communication between the autonomous vehicle 101 and external systems (such as devices, sensors, other vehicles, etc.). For example, the wireless communication system 112 can wirelessly communicate with one or more devices directly or via a communication network (such as servers 103-104 through the network 102). The wireless communication system 112 can use any cellular communication network or wireless local area network (WLAN), such as using WiFi to communicate with another component or system. The wireless communication system 112 can communicate directly with a device (e.g., a passenger's mobile device, a display device within the vehicle 101, a speaker) using, for example, an infrared link, Bluetooth, etc. The user interface system 113 can be part of a peripheral device implemented within the vehicle 101, including, for example, a keyboard, a touch screen display, a microphone, and a speaker.

[0043] Some or all of the functions of autonomous vehicle 101 may be controlled or managed by perception and planning system 110, particularly when operating in an autonomous driving mode. Perception and planning system 110 includes the necessary hardware (e.g., processor(s), memory, storage) and software (e.g., operating system, planning and routing programs) to receive information from sensor system 115, control system 111, wireless communication system 112, and / or user interface system 113, process the received information, plan a route or path from an origin to a destination point, and then drive vehicle 101 based on the planning and control information. Alternatively, perception and planning system 110 may be integrated with vehicle control system 111.

[0044] For example, a user as a passenger may specify a starting location and a destination of a trip, for example, via a user interface. The perception and planning system 110 obtains data related to the trip. For example, the perception and planning system 110 may obtain location and route information from an MPOI server (which may be part of the servers 103-104). The location server provides location services, and the MPOI server provides map services and POIs for certain locations. Alternatively, such location and MPOI information may be locally cached in a permanent storage device of the perception and planning system 110.

[0045] As the autonomous vehicle 101 moves along the route, the perception and planning system 110 may also obtain real-time traffic information from a traffic information system or server (TIS). Note that the servers 103-104 may be operated by a third-party entity. Alternatively, the functionality of the servers 103-104 may be integrated with the perception and planning system 110. Based on the real-time traffic information, MPOI information, and location information detected or sensed by the sensor system 115, and real-time traffic environment data (e.g., obstacles, objects, nearby vehicles), the perception and planning system 110 may plan an optimal route and, for example, drive the vehicle 101 via the control system 111 according to the planned route to reach a designated destination safely and efficiently.

[0046] The server 103 may be a data analysis system that performs data analysis services for various clients. In one embodiment, the data analysis system 103 includes a data collector 121 and a machine learning engine 122. The data collector 121 collects driving statistics 123 from various vehicles (autonomous vehicles or conventional vehicles driven by human drivers). The driving statistics 123 include information indicating driving commands issued (e.g., throttle, brake, steering commands) and responses of the vehicle captured by the vehicle's sensors at different time points (e.g., speed, acceleration, deceleration, direction). The driving statistics 123 may also include information describing the driving environment at different time points, such as, for example, routes (including starting locations and destination locations), MPOI, road conditions, weather conditions, etc.

[0047] Based on the driving statistics 123, the machine learning engine 122 generates or trains a set of rules, algorithms, and / or prediction models 124 for various purposes. In one embodiment, the algorithms 124 may include an algorithm or model that determines the evolution of the obstacle state of the spatial position of the moving obstacle over a period of time, an algorithm that determines the evolution of the lane-obstacle relationship between the moving obstacle and each of the one or more lanes near the moving obstacle over a period of time, and / or an algorithm or prediction model that predicts the expected movement of the moving obstacle based on the obstacle state evolution and the lane-obstacle evolution, which will be described in detail below. The algorithms 124 can then be uploaded to the ADV for real-time use during autonomous driving.

[0048] Figure 3A and Figure 3B is a block diagram illustrating an example of a perception and planning system for use with an autonomous vehicle according to one embodiment. System 300 may be implemented as Figure 1 A portion of an autonomous vehicle 101 includes, but is not limited to, a perception and planning system 110, a control system 111, and a sensor system 115. Figure 3A-3BThe perception and planning system 110 includes but is not limited to a positioning module 301 , a perception module 302 , a prediction module 303 , a decision module 304 , a planning module 305 , a control module 306 and a routing module 307 .

[0049] Some or all of the modules 301-307 may be implemented in software, hardware, or a combination thereof. For example, these modules may be installed in permanent storage device 352, loaded into memory 351, and executed by one or more processors (not shown). Note that some or all of these modules may be communicatively coupled to Figure 2 Some or all modules of the vehicle control system 111 or integrated with these modules. Some of the modules 301-307 can be integrated together as integrated modules.

[0050] The positioning module 301 determines the current location of the autonomous vehicle 300 (e.g., using the GPS unit 212) and manages any data related to the user's trip or route. The positioning module 301 (also known as the map and route module) manages any data related to the user's trip or route. The user can log in and specify the starting location and destination of the trip, for example, via a user interface. The positioning module 301 communicates with other components of the autonomous vehicle 300, such as map and route information 311, to obtain data related to the trip. For example, the positioning module 301 can obtain location and route information from a location server and a map and POI (MPOI) server. The location server provides location services, and the MPOI server provides map services and POIs for certain locations, which can be cached as part of the map and route information 311. As the autonomous vehicle 300 moves along the route, the positioning module 301 can also obtain real-time traffic information from a traffic information system or server.

[0051] Based on the sensor data provided by the sensor system 115 and the positioning information obtained by the positioning module 301, the perception of the surrounding environment is determined by the perception module 302. The perception information can represent what an average driver would perceive around the vehicle that the driver is driving. The perception can include lane configuration, traffic lights, the relative position of another vehicle, pedestrians, buildings, crosswalks, or other traffic-related signs (e.g., stop signs, give way signs), for example, in the form of objects. The lane configuration includes information describing one or more lanes, such as, for example, the shape of the lane (e.g., straight or curved), the width of the lane, how many lanes there are in the road, one-way or two-way lanes, merging or splitting lanes, exit lanes, etc.

[0052] The perception module 302 may include a computer vision system or functionality of a computer vision system to process and analyze images captured by one or more cameras in order to identify objects and / or features in the environment of the autonomous vehicle. Objects may include traffic signs, road boundaries, other vehicles, pedestrians, and / or obstacles, among others. The computer vision system may use object recognition algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system may map the environment, track objects, and estimate the speed of objects, among others. The perception module 302 may also detect objects based on other sensor data provided by other sensors such as radar and / or LIDAR.

[0053] For each object, the prediction module 303 predicts the behavior that the object will exhibit in this situation. The prediction is made based on the perception data of the perceived driving environment at the time according to a set of map / route information 311 and traffic rules 312. For example, if the object is a vehicle in the opposite direction and the current driving environment includes an intersection, the prediction module 303 will predict whether the vehicle is likely to move straight forward or turn. If the perception data indicates that there is no traffic light at the intersection, the prediction module 303 can predict that the vehicle may have to stop completely before entering the intersection. If the perception data indicates that the vehicle is currently in a lane that can only turn left or a lane that can only turn right, the prediction module 303 can predict that the vehicle will be more likely to turn left or right, respectively.

[0054] For each object, decision module 304 makes a decision about how to handle the object. For example, for a particular object (e.g., another vehicle in a cross-road) and metadata describing the object (e.g., speed, direction, turn angle), decision module 304 determines how to encounter the object (e.g., overtake, yield, stop, pass). Decision module 304 can make such a decision based on a set of rules (such as traffic rules or driving rules 312) that can be stored in persistent storage device 352.

[0055] The routing module 307 is configured to provide one or more routes or paths from the starting point to the destination point. For a given trip from the starting location to the destination location, for example, received from a user, the routing module 307 obtains route and map information 311 and determines all possible routes or paths from the starting location to the destination location. The routing module 307 can generate a reference line in a topographical map for each route (which determines the arrival at the destination location from the starting location). The reference line refers to an ideal route or path that is not interfered by other factors such as other vehicles, obstacles or traffic conditions. That is, if there are no other vehicles, pedestrians or obstacles on the road, the ADV should accurately or closely follow the reference line. The topographic map is then provided to the decision module 304 and / or the planning module 305. Based on other data provided by other modules (such as traffic conditions from the positioning module 301, the driving environment perceived by the perception module 302, and traffic conditions predicted by the prediction module 303), the decision module 304 and / or the planning module 305 check all possible routes to select and modify one of the best routes. The actual path or route used to control the ADV may be close to or different from the reference line provided by the routing module 307, depending on the specific driving environment at the time.

[0056] Based on the decision for each perceived object, the planning module 305 uses the reference line provided by the routing module 307 as a basis to plan a path or route and driving parameters (e.g., distance, speed, and / or turning angle) for the autonomous vehicle. That is, for a given object, the decision module 304 decides how to handle the object, and the planning module 305 determines how to perform the processing. For example, for a given object, the decision module 304 may decide to pass through the object, and the planning module 305 may determine whether to pass on the left or right side of the object. Planning and control data are generated by the planning module 305, and the planning and control data include information describing how the vehicle 300 moves in the next movement cycle (e.g., the next route / path segment). For example, the planning and control data may indicate that the vehicle 300 moves 10 meters at a speed of 30 miles per hour (mph) and then changes to the right lane at a speed of 25mph.

[0057] Based on the planning and control data, the control module 306 controls and drives the autonomous vehicle according to the route or path defined by the planning and control data by sending appropriate commands or signals to the vehicle control system 111. The planning and control data includes sufficient information to drive the vehicle from a first point on the route or path to a second point using appropriate vehicle settings or driving parameters (e.g., throttle, brake, steering commands) at different points in time along the route or path.

[0058] In one embodiment, the planning phase is performed in several planning cycles (also referred to as driving cycles), such as, for example, in time intervals of every 100 milliseconds (ms). For each planning cycle or driving cycle, one or more control commands will be issued based on the planning and control data. That is, every 100ms, the planning module 305 plans the next route segment or path segment (e.g., including the target position and the time required for the ADV to reach the target position). Alternatively, the planning module 305 can also specify a specific speed, direction and / or steering angle, etc. In one embodiment, the planning module 305 plans the route segment or path segment for the next predetermined time period (such as 5 seconds). For each planning cycle, the planning module 305 plans the target position for the current cycle (e.g., the next 5 seconds) based on the target position planned in the previous cycle. Then, the control module 306 generates one or more control commands (e.g., throttle, brake, steering control commands) based on the planning and control data of the current cycle.

[0059] Note that decision module 304 and planning module 305 can be integrated as an integrated module. Decision module 304 / planning module 305 can include a navigation system or the functionality of a navigation system to determine a travel path for an autonomous vehicle. For example, the navigation system can determine a range of speeds and directional headings to affect the movement of the autonomous vehicle along a path that substantially avoids perceived obstacles while generally moving the autonomous vehicle along a road-based path to a final destination. The destination can be set based on user input via user interface system 113. The navigation system can dynamically update the travel path while operating the autonomous vehicle. The navigation system can incorporate data from a GPS system and one or more maps to determine the travel path of the autonomous vehicle.

[0060] Although not shown, the perception and planning system 110 also includes a data recorder or data collector configured to collect data processed by the components 301-307 and sensors over a period of time, which can be stored as part of the data record 313. For example, the data record 313 includes any predictions, decisions, and paths planned or made during each driving / planning cycle. The data record 313 may also include control commands issued, as well as the response or state of the vehicle captured at different time points in each driving cycle (e.g., speed, acceleration, heading, position, etc.). The data record 313 may also include moving obstacles perceived over a period of time and their behavior (e.g., previous position, speed, heading, etc. captured during past cycles). The data record 313 can be subsequently utilized to plan and control the vehicle, or alternatively, the data record 313 can be analyzed offline to improve the driving algorithm or prediction model.

[0061] According to one embodiment, the prediction module 303 is configured to identify moving obstacles based on the perception information provided by the perception module 302. For each of the moving obstacles in the ADV vicinity, the prediction module 303 predicts the expected movement of the moving obstacle based on the obstacle state evolution and the lane-obstacle evolution and based on a set of lane attention prediction rules or models 314.

[0062] Figure 4 is a block diagram showing an example of a prediction module according to one embodiment, and Figure 5 is a processing flow chart of a prediction module according to one embodiment. Figure 4 and Figure 5 , the prediction module 303 includes but is not limited to a spatiotemporal (ST) graph generator 401, a state evolution generator 402, a lane-obstacle relationship evolution generator 403, a spatial aggregation determination module 404, and a trajectory predictor 404, which work together using a lane attention prediction algorithm or model 314 to determine or predict the expected movement of a mobile obstacle based on the obstacle state evolution and the lane-obstacle evolution. Note that modules 401-405 can be integrated into a smaller number of modules or a single module. A mobile obstacle can be a vehicle, a pedestrian, a bicycle, or any other object capable of moving.

[0063] According to one embodiment, a moving obstacle in the vicinity of the ADV and one or more lanes around the moving obstacle can be identified. The moving obstacle can be a vehicle, a motorcycle, a bicycle, or a pedestrian. The moving obstacle can be detected by the perception module 302 based on various sensors installed on the ADV (such as Figure 2 The ST diagram generator 401 is configured to generate an ST diagram based on the sensor data obtained by the sensor (shown in Figure 4) to identify the mobile obstacle and one or more lanes around the mobile obstacle. In one embodiment, based on the perception information, the ST diagram generator 401 is configured to generate an ST diagram based on the spatial position of the mobile obstacle and the lane-obstacle relationship evolution between the mobile obstacle and each of the one or more lanes. The state evolution generator 402 is configured to determine the obstacle state evolution of the spatial position of the mobile obstacle over a period of time, and the lane-obstacle relationship evolution generator 403 is configured to determine the lane-obstacle relationship evolution between the mobile obstacle and each of the one or more lanes near the mobile obstacle over a period of time. The trajectory predictor 405 is configured to predict the expected movement of the mobile obstacle based on the obstacle state evolution and the lane-obstacle evolution. Thereafter, the planning module 305 is configured to plan the trajectory of the ADV based on the predicted expected movement of the mobile obstacle to control the ADV to avoid collision with the mobile obstacle.

[0064] In one embodiment, the spatial aggregation determination module 404 is configured to determine an aggregated lane code indicating the expected movement of the moving obstacle, wherein the expected movement of the moving obstacle is also predicted based on the aggregated lane code. In one embodiment, for each lane in the one or more lanes, the spatial aggregation determination module 404 is configured to determine the lane code based on the evolution of the lane-obstacle relationship of the moving obstacle to the lane, the current relative position of the lane to the moving obstacle, and the future shape of the lane. In one embodiment, for each lane in the one or more lanes, the spatial aggregation determination module 404 is configured to determine the attention score of the lane based on the evolution of the lane-obstacle relationship of the moving obstacle to the lane and the current relative position of the lane to the moving obstacle. The spatial aggregation determination module 404 is configured to determine the aggregated lane code based on the weighted sum of the lane codes of each lane in the one or more lanes, wherein the weight of each lane in the one or more lanes is based on the attention score of the lane.

[0065] In one embodiment, the lane attention prediction algorithm or model 314 can be based on a long short-term memory (LSTM) network and a multi-layer perception (MLP) network. For example, the obstacle state evolution of the spatial position of a moving obstacle over a period of time can be determined by using an LSTM network and an MLP network; the lane-obstacle relationship evolution of the moving obstacle to each lane in one or more lanes over a period of time can be determined by using another LSTM network and another MLP network.

[0066] Figure 6 600 is a diagram showing an example of the problem of predicting the trajectory of a moving obstacle 601 having one or more lanes (e.g., 602, 603, 604) around the moving obstacle 601. In this example, the obstacle 601 is moving in a specific direction at a specific speed near an exit, where there are one or more lanes (e.g., 602, 603, 604) near the moving obstacle 601. The moving obstacle 601 can be constrained to drive on one of the one or more lanes (e.g., 602, 603, 604). When planning a path for an ADV 605, it is important to accurately predict which lane of the lanes (e.g., 602, 603, 604) the moving obstacle 601 will most likely take. Which lane the moving obstacle 601 will take may greatly affect how the ADV 605 will move to avoid colliding with the moving obstacle 601.

[0067] Fig. 7A700a is a diagram showing an example of predicting pedestrian trajectories based on dynamic interactions by a grid-like Euclidean method. The Euclidean method has been used to learn the dynamic interactions of pedestrians and use the dynamic interactions to guide the prediction of pedestrian trajectories. For example, the Euclidean method has been used to predict the trajectories of pedestrians in crowded scenes. In the Euclidean method, the space is divided into grids and occupancy grid pooling or social pooling is applied. However, this method does not take into account the attention of pedestrians. Figure 7B 700b is a diagram showing an example of predicting a pedestrian's trajectory based on dynamic interactions by a non-Euclidean method. Non-Euclidean methods have been used to treat objects and their interactions as graphs and predict pedestrian trajectories using an attention mechanism.

[0068] But if combined Figure 6 As discussed, the behavior of a moving obstacle may be constrained by the lane information rather than by the dynamics of the moving obstacle or the accidental interactions with other moving obstacles. Therefore, the influence of the static environment may dominate when determining the future movement trajectory of the moving obstacle.

[0069] Figure 7C 700c is a diagram showing an example of predicting the trajectory of a moving obstacle (e.g., a moving vehicle) based on a static environment by a grid-like Euclidean method. The Euclidean method treats the road infrastructure as Euclidean data (e.g., a semantic graph). However, the structure of the lanes on the road may not be uniform. There can be any number of lanes around a moving obstacle, ranging from one to a large number (e.g., when entering a large intersection with many branches). In addition, the shape or direction of the lanes may vary: on a highway, the lanes are mostly straight; while within an intersection, the lanes may branch into several completely different directions. When driving, the driver may focus on one or a few lanes based on his or her intended intention to move. In some cases, the driver may tend to follow the direction of the lane. Because the Euclidean method does not take the above factors into account, the Euclidean method may not accurately predict the trajectory of a moving obstacle. It may be necessary to develop a method or model to accurately predict the trajectory of a moving obstacle based on the expected movement of the moving obstacle.

[0070] Fig.7D700d is a diagram showing an example of predicting the trajectory of a moving obstacle based on a static environment according to one embodiment. The paired relationship between a moving obstacle (e.g., 701) and its surrounding lanes (e.g., 702, 703, 704) plays an important role in predicting the future movement of the moving obstacle. Therefore, the method for predicting the trajectory of a moving obstacle can be based on lane information (e.g., 702, 703, 704) and an attention mechanism to aggregate static environment information. In this method, the relationship between the moving obstacle and the surrounding lanes can be learned. By understanding and modeling the influence from the surrounding environment, the method can accurately predict the trajectory or expected movement of the moving obstacle. In addition, the method can be applied to both autonomous driving based on high-definition (HD) maps and autonomous driving based on non-HD maps. In autonomous driving based on HD maps, lane information can be provided by pre-collected HD maps. In autonomous driving based on non-HD maps, lanes detected by cameras or pre-collected human driving paths can be used as lane information. In addition, because the learned attention scores can be visualized, the method can provide an intuitive explanation of the behavior of the moving obstacle. This interpretability can also benefit other downstream modules of the autonomous driving system.

[0071] Figure 8 800 is a diagram illustrating an example of a method for predicting a trajectory of a moving obstacle 801 within an ADV vicinity (not shown), where the moving obstacle 801 may be near one or more lanes 802, 803, 804. In one embodiment, the method may predict the trajectory of the moving obstacle 801 by formulating an ST graph 805 for unfolding the motion history of the moving obstacle on the one or more lanes 802, 803, 804. The moving obstacle 801 may be, for example, a moving vehicle. Figure 8 As shown, one or more lanes 802, 803, 804 may be near an exit.

[0072] In this method, the sensor of the perception module of the ADV can receive each moving obstacle from the previous time -T in increments of Δt (the sampling period of the sensor). obs to the historical position at the current time step t = 0. It can be assumed that the surrounding lanes of the moving obstacle are given at each time step, and the number of lanes is represented as N. Through this method, it is possible to predict the position of each vehicle at T pred Future position in time span, T pred It can be an integer multiple of Δt. For example, the method can be used to predict the moving obstacle 801 at T pred Future position over a time span.

[0073] The static environment of the mobile obstacle 801 can be modeled by using a graph neural network (GNN). A GNN-based ST graph 805 can be generated to clearly indicate the pairwise relationship between the mobile obstacle 801 and one or more lanes 802, 803, 804. GNN is a connectionist model that captures the dependencies of a graph via messages passed between the nodes of the graph. Unlike standard neural networks, GNNs can retain states that can represent information from neighborhoods with arbitrary depth. GNNs are very effective in processing non-Euclidean inputs or mining pairwise relationship features from input data. The spatiotemporal graph neural network (ST-GNN) is a derivative of GNN, using nodes to represent entities and two types of edges to represent temporal and spatial relationships.

[0074] like Figure 8 As shown, for a moving obstacle 801, an ST graph 805 can be generated: G = (ν; ε T ; ε S ), where ν is the set of nodes, ε T is the set of temporal edges, and ε S is a set of spatial edges, as described in equations (1) and (2) below:

[0075] and in

[0076]

[0077]

[0078]

[0079] As shown in equation (1), the node set ν contains two types of nodes: obstacle nodes and lane nodes. For example, the obstacle node v t 811 represents a moving obstacle 801 at a given time t, and lane nodes (e.g., 821, 822, 823) represents one of the local lanes (e.g., 802, 803, 804) around the moving obstacle 801 at time t. Obstacle node v t+1 811′ represents the moving obstacle 801 at time step t+1, and the lane node (e.g., 821′, 822′, 823′) represents one of the local lanes (e.g., 802, 803, 804) around the moving obstacle at time step t+1, and so on.

[0080] Can be based on the obstacle node v t 811 and lane nodes The pairwise relationships (e.g. 821, 822, 823) at the same time t form a set of spatial edges ε S (For example 831, 832, 833). Figure 8 As shown, at a given time t, the connecting line 831, 832 and 833 represent obstacle nodes v t 811 and lane nodes 821, 822 and 823. At time step t+1, the connecting line 831′ 832′ and 833′ respectively represent obstacle nodes v t+1 811′ and lane nodes 821′ 822′ and 823′ between the space edges.

[0081] As shown in Equation (2), there are two types of temporal edges, one type is about the evolution of obstacle states, and the other type is about the evolution of lane-obstacle relationships over time. t 811 and obstacle node v t+1 The temporal edge 841 of 811′ represents the obstacle state evolution. A set of spatial edges ε at a given time t S (For example 831, 832, 833) and a set of spatial edges ε at time step t+1 S (For example 831′ 832′ and 833′) can represent the evolution of the lane-obstacle relationship over time. The movement history of the mobile obstacle 801 and its changing relationship with the surrounding lanes 802, 803, 804 can be expanded over time to form an ST graph 805, such as Figure 8 shown.

[0082] Figure 9A-9E Show Figure 8Details of an example of a method in which the trajectory of a moving obstacle 801 within the vicinity of an ADV is predicted. Fig.9A FIG900a is a diagram showing the steps of receiving new information about a moving obstacle 801. Figure 8 and Fig.9A , the spatial position of the moving obstacle 801 can be received at each moment. Lane information about one or more lanes (e.g., 802, 803, 804) near the moving obstacle 801 can also be received at each moment. For example, at each moment, the obstacle node v t 811 can receive the new spatial position of the moving obstacle 801 And you can also refresh the lane node (e.g., 821, 822, 823) to reflect the lanes (e.g., 802, 803, 804) in the current neighborhood of the moving obstacle 801. Lane nodes (e.g., 821, 822, 823) may include a set of ordered lane points. Lane information about lanes (e.g., 802, 803, 804) may come directly from the sensor of the perception module of the ADV. Alternatively, information about lanes (e.g., 802, 803, 804) may be obtained by first locating the position of the moving obstacle 801 and then extracting information about lanes (e.g., 802, 803, 804) around the moving obstacle 801 from a pre-collected HD map. Therefore, all spatial edges at this moment are (For example 831, 832, 833) can be easily updated with the new spatial relationship of the moving obstacle 801 to the local lanes (e.g., 802, 803, 804) near the moving obstacle.

[0083] Fig. 9B 900b is a diagram showing the steps of the time evolution of the method for predicting the trajectory of the mobile obstacle 801. For example, the time evolution may include the obstacle state evolution of the spatial position of the mobile obstacle 801 and the lane-obstacle relationship evolution of the mobile obstacle 801 and each of the one or more lanes (e.g., 802, 803, 804) near the mobile obstacle 801 over a period of time. For example, the method may update the time edge 851′ representing the obstacle state evolution of the mobile obstacle 801 and a set of time edges (e.g., 852′) representing the lane-obstacle relationship evolution of the mobile obstacle 801 and the one or more lanes (e.g., 802, 803, 804).

[0084] In one embodiment, the obstacle state evolution can be determined by using a long short-term memory network (LSTM). LSTM is an artificial recurrent neural network (RNN) architecture used in the field of deep learning. Unlike standard feedforward neural networks, LSTM has feedback connections. LSTM can process not only a single data point, but also an entire data sequence. LSTM can be applied to learn patterns in sequence data. For example, the LSTM network can be described by the following equation:

[0085] f t =σ(W f x t +U f h t-1 +b f ) (3)

[0086] i t =σ(W i x t +U i h t-1 +b i ) (4)

[0087] o t =σ(W o x t +U o h t-1 +b o ) (5)

[0088] c t =f t ⊙c t-1 +i t ⊙tanh(W c x t +U c h t-1 +b c ) (6)

[0089] h t =o t ⊙tanh(c t ) (7)

[0090] In the above equation, f t 、i t , o t and c t Respectively represent the forget gate, update gate, output gate and unit state, h t is a hidden state and may contain the encoded pattern of the sequence input. The following equation (8) can be used as an abbreviation of equations (3)–(7):

[0091] h t=LSTM(h t-1 ,x t ; θ) (8)

[0092] The obstacle movement is in the form of sequence data, and the movement may be partially governed by kinematics and vehicle dynamics, especially in the short term. For example, a vehicle cannot make a sharp turn in an instant; nor can a vehicle decelerate from 60 mph to 0 mph in the blink of an eye. Therefore, the LSTM network can be used to learn this potential driving force. For example, the LSTM network can first embed the relative displacement of the spatial position of the moving obstacle 801 using a multi-layer perceptron (MLP) network, as described in the following equation (9). The MLP network is a type of feedforward artificial neural network. The MLP network may include at least three layers of nodes: an input layer, a hidden layer, and an output layer. The MLP can be trained using a supervised learning technique called backpropagation. Next, the embedded and previously hidden states can be used as input to update the temporal edge 851′(v t-1 , v t )'s new hidden state 901, as described in equation (10) below.

[0093]

[0094]

[0095] The evolution of the lane-obstacle relationship of the moving obstacle 801 can be determined by using another LSTM network. The movement of the obstacle can be determined by the driver's intention. The driver's intention may not usually be explicitly expressed, but can be inferred based on the changing relationship of the obstacle to each lane, because the driver tends to follow one or several lanes to maintain courtesy and avoid accidents. Therefore, the evolution of the lane-obstacle relationship of the moving obstacle 801 and each of the one or more lanes (e.g., 802, 803, 804) over a period of time can be determined by using another LSTM network.

[0096] refer to Figure 8 and Fig. 9B , using the spatial position of the moving obstacle 801 811 and updated local lane information (e.g., 821, 822, 823), the position of the mobile obstacle 801 on each lane (e.g., 802, 803, or 804) can be projected to obtain a projection point Then, the difference between the projected point and the position of the moving obstacle 801 can be determined, and this vector can be embedded using an MLP, as described in equation (11) below. Next, this embedding and the previously hidden state can be used to update the temporal edges corresponding to the connecting sequential lane-vehicle relationship pairs, as shown in equation (12) below. (e.g. 852') (e.g. 911).

[0097]

[0098]

[0099] As shown in equations (11) and (12), the encoding (e.g., 911) includes the learned evolution relationship between the moving obstacle 801 and the i-th lane of one or more lanes (e.g., 802, 803, 804). 911 includes the learned evolution relationship between the moving obstacle 801 and the first lane 802, such as Fig. 9B shown.

[0100] Fig. 9C 900c is a diagram showing the steps of spatial aggregation of the method for predicting the trajectory of the moving obstacle 801. Spatial aggregation refers to the aggregation of information about the obstacle state of the moving obstacle 801 and lane information about all lanes (e.g., 802, 803, 804) to infer the intention of the driver of the moving obstacle 801 so as to accurately predict the further trajectory of the moving obstacle 801. As described above, for each lane, the encoding of the historical evolution relationship between the lane and the moving obstacle 801 can be determined by using an LSTM network. (e.g., 911). Next, the current relative position of each lane to the moving obstacle 801 and the future shape of each lane can be determined using the MLP network, for example, as described in the following equations (13) and (14), respectively:

[0101]

[0102]

[0103] Each lane can be coded The current relative position of each lane and the future shape of each lane are concatenated together to form the total encoding for each lane at a given time t

[0104]

[0105] For example, Figure 8 and Fig. 9C As shown, for the first lane 802, the encoding 911. Current relative position 912 and future shapes 913 are connected together to form the total code at a given time t 921. Similarly, for the second lane 803, the total code at a given time t can be formed 922; For the third lane 804, the total code at a given time t can be formed 923.

[0106] In order to accurately predict the trajectory of the moving obstacle 801, the lane information of all surrounding lanes (e.g., 802, 803, 804) needs to be considered. In order to jointly reason across multiple lanes, the encodings of all lanes (e.g., 921, 922, 923) can be aggregated. This is a challenging task because there may be a variable number of lanes, but the aggregated output should be compact and fixed in size. Moreover, different lanes can play different roles in determining the movement of the moving obstacle 801.

[0107] For example, a lane merging mechanism may be used to aggregate all lanes. In the lane merging mechanism, the determining factor may be based on a single lane. The single lane is the lane closest to the moving obstacle 801, and the lane may change over time. At each time step, the encoding of the lane closest to the moving obstacle 801 may be selected as the lane merging result. The selected encoding may be used as the aggregate encoding a t :

[0108]

[0109]

[0110] However, the driver of the mobile obstacle 801 may not only focus on a single lane when driving; the driver may prefer to pay attention to multiple lanes. In addition, in some cases (such as in the middle of a lane change behavior), the lane merging result may change suddenly, which may have some negative effects on subsequent network modules.

[0111] Fig.9D FIG. 8 shows a prediction of the trajectory of a moving obstacle 801 based on a lane attention mechanism according to an embodiment. The lane attention mechanism can be used to solve the above problems. The lane attention mechanism refers to a prediction algorithm or model that aggregates the encoding of all lanes based on the driver's attention to each lane separately. For example, the lane attention mechanism can be implemented by the lane attention prediction model 314 of the ADV. In one embodiment, first, based on the current relative position of each lane, and the encoding of the historical evolution relationship of each lane to the moving obstacle 801 Calculate the attention score for each lane,

[0112]

[0113] refer to Fig.9D , based on the current relative position of the first lane 912 and the first lane's encoding of the evolving relationship of the moving obstacle 801 911, calculating an attention score for the first lane. The attention score for each lane indicates the driver's attention to each lane, which corresponds to the possibility that a moving obstacle may be driven in the corresponding lane.

[0114] Then, the total encoding per lane can be obtained by taking the total encoding per lane from equation (15) The total aggregate code is calculated by weighted sum of t 930, where the weight is the normalized attention score as described in equation (19) below,

[0115]

[0116] It can be expected that the aggregate lane encoding a obtained based on the lane merging mechanism or the lane attention mechanism t 930 contains the learned encoding of the driver's intention to move the obstacle 801. Aggregate lane encoding a based on lane attention mechanism t The 930 can accurately predict the driver's intentions by taking into account the driver's attention in all lanes.

[0117] Fig.9E 801. The aggregate lane encoding and the previous encoding of the obstacle's movement history can be combined and used to update the state corresponding to the obstacle node v t The total hidden state of:

[0118]

[0119]

[0120] like Fig.9E As shown, the total hidden state can be updated at each time step and can be used to infer the future trajectory of the moving obstacle 801.

[0121] When at time t (where t∈[1,T pred / Δt]) When predicting the trajectory of the moving obstacle 801, it can be assumed that each trajectory point follows a bivariate Gaussian distribution, and the network can be trained to learn all the parameters of the Gaussian distribution. Therefore, the hidden state of the obstacle node can be processed using an MLP with the last rectified linear unit (ReLU) layer removed. A 5-dimensional vector may be determined for each trajectory point, which may include the value of the mean vector and the covariance matrix, as described in the following equation (22):

[0122]

[0123] Expected value of the forecast distribution Can replace The new spatial position of the moving obstacle 801 is used as the input of the next cycle LSTM and the trajectory point of the next time step is inferred. This process can be repeated until t=T is predicted. pred / Δt for all trajectory points.

[0124] For example, the negative log-likelihood can be used as a loss function, and the network can be trained by minimizing this loss function, as described in the following equation (23):

[0125]

[0126] Fig.10 1000 is a flow chart illustrating a process of predicting a trajectory of a moving obstacle according to one embodiment. Process 1000 may be performed by processing logic, which may include software, hardware, or a combination thereof. For example, process 1000 may be performed by prediction module 303. Fig.10 In operation 1001, the processing logic may generate a spatiotemporal graph based on the spatial position of the mobile obstacle and the evolution of the lane-obstacle relationship between the mobile obstacle and each of one or more lanes. In operation 1002, the processing logic determines the evolution of the obstacle state of the spatial position of the mobile obstacle over a period of time. In operation 1003, the processing logic determines the evolution of the lane-obstacle relationship between the mobile obstacle and each of one or more lanes near the mobile obstacle over a period of time. In operation 1004, the processing logic may determine an aggregate lane code indicating the expected movement of the mobile obstacle. In operation 1005, the processing logic predicts the expected movement of the mobile obstacle based on the obstacle state evolution and the lane-obstacle evolution. In one embodiment, the expected movement of the mobile obstacle may also be predicted based on the aggregate lane code. In operation 1006, the processing logic plans the trajectory of the ADV based on the predicted expected movement of the mobile obstacle to control the ADV to avoid collision with the mobile obstacle.

[0127] Note that some or all of the components shown and described above may be implemented in software, hardware, or a combination thereof. For example, such components may be implemented as software installed and stored in a permanent storage device, which may be loaded by a processor (not shown) and executed in a memory to implement the processes or operations described throughout this application. Alternatively, such components may be implemented as executable code programmed or embedded in dedicated hardware such as an integrated circuit (e.g., a dedicated IC or ASIC), a digital signal processor (DSP), or a field programmable gate array (FPGA), which may be accessed from an application via a corresponding driver and / or an operating system. In addition, such components may be implemented as specific hardware logic in a processor or processor core as part of an instruction set accessible to the software component via one or more specific instructions.

[0128] Some portions of the preceding detailed description have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a desired result. These operations are those requiring physical manipulations of physical quantities.

[0129] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise clearly indicated from the foregoing discussion, it should be understood that throughout this specification, discussions utilizing terms such as those set forth in the appended claims refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within computer system registers and memories into other data similarly represented as physical quantities within computer system memories or registers or other such information storage, transmission or display devices.

[0130] Embodiments of the present disclosure also relate to apparatus for performing the operations herein. Such a computer program is stored in a non-transitory computer-readable medium. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). For example, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) readable storage medium (e.g., a read-only memory ("ROM"), a random access memory ("RAM"), a magnetic disk storage medium, an optical storage medium, a flash memory device).

[0131] The processes or methods depicted in the foregoing figures may be performed by processing logic including hardware (e.g., circuits, dedicated logic, etc.), software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. Although the processes or methods are described above in terms of some sequential operations, it should be understood that some of the operations described may be performed in a different order. In addition, some operations may be performed in parallel rather than sequentially.

[0132] The embodiments of the present disclosure are not described with reference to any particular programming language. It will be appreciated that a variety of programming languages ​​may be used to implement the teachings of the embodiments of the present disclosure as described herein.

[0133] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments of the present disclosure. Obviously, various modifications may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the appended claims. Therefore, the description and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A computer-implemented method for operating an autonomous vehicle (ADV), the method comprising: Determine the spatial position of the moving obstacle and the evolution of the obstacle state over a period of time; determining an evolution of a lane-obstacle relationship between a moving obstacle and each of one or more lanes in the vicinity of the moving obstacle over a period of time; Predict the expected movement of moving obstacles based on the evolution of obstacle states and the evolution of lane-obstacle relationships; as well as planning a trajectory of the ADV to control the ADV to avoid collision with the moving obstacle based on the predicted expected movement of the moving obstacle; Predict the expected movement of moving obstacles based on the evolution of obstacle states and lane-obstacle relationships, including: For each lane of the one or more lanes, determining a lane coding based on an evolution of a lane-obstacle relationship between the moving obstacle and the lane, a current relative position of the lane to the moving obstacle, and a subsequent shape of the lane; For each lane of the one or more lanes, determining an attention score for the lane based on a lane-obstacle state evolution of the moving obstacle and the lane and a current relative position of the lane to the moving obstacle; determining an aggregate lane coding based on a weighted sum of lane codings for each of the one or more lanes, wherein a weight for each of the one or more lanes is based on an attention score for the lane; Predict the expected movement of moving obstacles based on aggregated lane coding.

2. The method according to claim 1 further includes generating a spatiotemporal graph based on the spatial position of the mobile obstacle and the evolution of the lane-obstacle relationship between the mobile obstacle and each lane in one or more lanes.

3. The method according to claim 2, wherein the spatiotemporal graph includes multiple types of nodes, a set of spatial edges and multiple types of temporal edges.

4. The method according to claim 3, wherein the plurality of types of nodes include: obstacle nodes representing the spatial position of a moving obstacle at a given time; and a corresponding lane node representing each lane of the plurality of lanes at a given time. 5 . The method according to claim 4 , wherein each spatial edge in the set of spatial edges indicates a corresponding pairwise relationship between an obstacle node and a corresponding lane node.

6. The method according to claim 3, wherein the multiple types of time edges include: The obstacle state evolution of the spatial position of the moving obstacle over a period of time, and the lane-obstacle relationship evolution of the moving obstacle and each of the one or more lanes over a period of time.

7. The method of claim 1, wherein the obstacle state evolution of the spatial position of the mobile obstacle over a period of time is determined by using at least one of a long short-term memory (LSTM) network or a multi-layer perception (MLP) network.

8. The method of claim 1 , wherein the evolution of a lane-obstacle relationship of a moving obstacle to each of one or more lanes over a period of time is determined by using at least one of a long short-term memory (LSTM) network or a multi-layer perceptron (MLP) network.

9. A non-transitory machine-readable medium having instructions stored therein that, when executed by a processor, cause the processor to perform the operations of the method of operating an autonomous vehicle (ADV) as claimed in any one of claims 1 to 8.

10. A data processing system comprising: processor; as well as A memory coupled to the processor and configured to store instructions that, when executed by the processor, cause the processor to perform the operations of the method of operating an autonomous vehicle (ADV) as claimed in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that The computer program, when executed by a processor, implements the operations of the method of operating an autonomous driving vehicle (ADV) as claimed in any one of claims 1 to 8.

Images