Method and system for pedestrian action prediction based on graph neural networks in autonomous driving systems

Abstract

The present invention relates to methods and systems for spatiotemporal graph modeling of road users in an observation frame (i.e. traffic scene) of an environment in which an autonomous vehicle operates, clustering the road users into several categories, and providing the spatiotemporal graph to a trained graphical neural network (GNN) to predict a future pedestrian action. The future pedestrian action can be one of: a pedestrian will cross the road and a pedestrian will not cross the road. The spatiotemporal graph includes a better understanding of the observation frame (i.e. traffic scene).

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 63 / 108,336, filed October 31, 2020, entitled “METHOD AND SYSTEM FOR GRAPH NEURAL NETWORKBASED PEDESTRIAN ACTION PREDICTION IN AUTONOMOUSDRIVING SYSTEMS,” the contents of which are incorporated herein by reference. Technical Field

[0003] This invention generally relates to methods and systems for predicting pedestrian actions in autonomous driving systems, and more specifically to graph neural network-based methods and systems for predicting pedestrian actions in autonomous driving systems. Background Technology

[0004] An autonomous vehicle (i.e., an autonomous car) includes an autonomous driving system (ADS) that controls the autonomous vehicle's automated operation in its environment. An autonomous vehicle includes different types of environmental sensors mounted on it (e.g., cameras such as optical cameras, and detection and ranging sensors such as light detection and ranging (LIDAR) sensors) that sense the environment around the autonomous vehicle. The ADS of an autonomous vehicle includes subsystems that process environmental sensor data received from different types of sensors (e.g., image data from cameras, 3D point clouds from detection and ranging sensors) to generate a map of the vehicle's surroundings that includes the vehicle and each road user in the environment. The ADS system also includes a planning subsystem that plans a geometric path for the autonomous vehicle and controls its operation based on the planned path.

[0005] ADS can also be one or more machine learning-based behavior prediction subsystems. For ADS, it is important to be able to generate predictions of how dynamic objects near the autonomous vehicle will behave (e.g., how a pedestrian will move while crossing the street) so that the desired behavior of the autonomous vehicle can be planned (e.g., to avoid colliding with pedestrians crossing the street). This type of subsystem is called a pedestrian action prediction (i.e., pedestrian intention prediction) subsystem. For example, the pedestrian action prediction subsystem can predict the trajectory of a pedestrian approaching the autonomous vehicle (e.g., within approximately 30 meters of the autonomous vehicle) and then predict the pedestrian's action based on the predicted trajectory. The predicted pedestrian action can be one of the following: (a) the pedestrian will cross in front of the autonomous vehicle, or (b) the pedestrian will not cross in front of the autonomous vehicle. Alternatively, the pedestrian action prediction subsystem can use a graph to model the interaction between the pedestrian and dynamic objects in the surrounding environment and use a graph neural network to predict the pedestrian's future behavior. The planning system then uses the predicted pedestrian actions to modify the autonomous vehicle's planned path to avoid collisions with pedestrians.

[0006] Predicting pedestrian actions is a complex, computationally intensive task because it requires modeling the interaction between pedestrians and their surrounding environment, which depends on various environmental factors and the spatiotemporal context of that environment. Therefore, there is a need to improve methods and systems for predicting pedestrian actions. Summary of the Invention

[0007] This invention describes exemplary methods and systems for: modeling a spatiotemporal graph of road users in an observation frame (i.e., a traffic scene) of an environment in which an autonomous vehicle is operating, clustering the road users into several categories, and feeding the spatiotemporal graph to a trained graphical neural network (GNN) to predict future pedestrian actions. The future pedestrian actions may be one of: a pedestrian crossing the road or a pedestrian not crossing the road.

[0008] According to one exemplary aspect, a computer-implemented method for predicting pedestrian actions is provided. The method includes receiving a time series of observation frames, each observation frame including spatial information of a target pedestrian and a plurality of road users. For each observation frame in the sequence of observation frames, the method includes: encoding a target pedestrian feature set for the target pedestrian and a corresponding road user feature set for each of the plurality of road users, at least based on the spatial information included in the observation frame; generating a set of relative importance weights, including a corresponding relative importance weight for each of the road users, the corresponding relative importance weight indicating the relative importance of the road user to the target pedestrian, the corresponding relative importance weight for each road user being based on the distance between the road user and the target pedestrian and the relative positional importance of the road user to the target pedestrian. The method further includes predicting the future actions of the target pedestrian based on the target pedestrian feature set encoded for each of the plurality of observation frames, the corresponding road user feature set encoded for each of the plurality of observation frames, and the relative importance weight set generated for each of the plurality of observation frames. The actions of autonomous vehicles can be automatically controlled based on the predicted future actions of the target pedestrian.

[0009] According to the above aspects of the computer implementation method, the relative positional importance of each road user can be based on the road user's direction of movement relative to the target pedestrian.

[0010] In one or more of the above aspects of the computer implementation method, the relative positional importance of each road user may be greater when the road user moves toward the target pedestrian than when the road user moves away from the target pedestrian.

[0011] In one or more of the above aspects of the computer implementation method, the relative positional importance of each road user may also be based on the distance traveled along the road by the road user relative to the position of the target pedestrian.

[0012] In one or more of the above aspects of the computer implementation method, the relative positional importance of each road user may be based on the distance between the road user and a reference line extending from the position of the target pedestrian and perpendicular to the direction of travel on the road.

[0013] In one or more of the above aspects of the computer implementation method, for each road user, the distance between the road user and the target pedestrian is a Euclidean distance.

[0014] In one or more of the above aspects of the computer implementation method, the computer implementation method includes, for each observation frame in the sequence of observation frames: encoding the target pedestrian feature set of the target pedestrian and a corresponding road user set of each of the plurality of road users based on the spatial information included in the plurality of observation frames in the sequence including the observation frame; generating the relative importance weight set for each road user based on the spatial information included in the plurality of observation frames in the sequence including the observation frame.

[0015] In one or more of the above aspects of the computer implementation method, the computer implementation method includes, for each observation frame in the sequence of observation frames: clustering the road user groups of the plurality of road users into corresponding clusters based on behavioral similarity, based on the spatial information included in the plurality of observation frames in the sequence including the observation frames; and predicting the future action of the target pedestrian also based on the clustering.

[0016] In one or more of the above aspects of the computer implementation method, a corresponding spatial graph is generated for each of the observation frames, wherein, for each observation frame: the corresponding spatial graph has a target pedestrian node representing the target pedestrian, and a plurality of road user nodes each representing a corresponding one of the plurality of road users, the corresponding spatial graph being defined by: (i) a feature matrix including the encoded target pedestrian features as features of the target pedestrian node, and including the set of road user features encoded for the corresponding road user as features of the corresponding road user node; (ii) an adjacency matrix specifying: (a) a corresponding weighted connection edge between the target pedestrian node and each of the corresponding road user nodes, the target pedestrian node and each of the corresponding road user nodes corresponding to the set of relative importance weights generated for the observation frame; (b) connection edges between each of the road user nodes included in the corresponding cluster.

[0017] In one or more of the above aspects of the computer implementation method, predicting the future action of the target pedestrian is performed using a spatiotemporal convolutional graph neural network that receives the spatial graph generated for the observation frame.

[0018] In one or more of the above aspects of the computer implementation method, the predicted pedestrian action is one of the pedestrian crossing in front of the autonomous vehicle or the pedestrian not crossing in front of the autonomous vehicle.

[0019] In one or more examples of the above aspects of the computer implementation method, the computer implementation method includes, for each observation frame in the sequence of observation frames: the corresponding set of road user features encoded for each of the plurality of road users includes one or more of the following: the type of the road user; the position of the road user relative to the target pedestrian; the size of the road user; the speed of the road user; and the direction of movement of the road user.

[0020] According to another exemplary aspect, a computer-implemented method for predicting pedestrian actions in an autonomous vehicle is provided. The computer-implemented method includes receiving a set of observation frames, each observation frame including a pedestrian and a plurality of road users, and generating spatiotemporal information about the pedestrian based on the set of observation frames, the spatiotemporal information including the position of the pedestrian in each of the observation frames. The method further includes generating spatial information about each road user in a map based on the set of observation frames, the spatial information including the position of each road user in the map, generating a feature representation of the pedestrian based on the spatiotemporal information about the pedestrian, and generating features encoded for each corresponding road user based on the spatiotemporal information about the corresponding road user. For each corresponding observation frame, the method further includes generating a spatial map based on the position of the pedestrian in the observation frame and the position of each road, the spatial map including a central node, a plurality of other nodes surrounding the central node, and connections between the central node and each of the plurality of other nodes, wherein the central node is associated with the pedestrian, and each of the other nodes is associated with one of the plurality of road users. The method further includes: generating an adjacency matrix for each spatial graph, the adjacency matrix of each spatial graph including, for each connection between the central node and one of the other nodes in the graph, a weight representing the importance of the road user associated with the other node; generating a feature encoding matrix including feature codes of the plurality of road users and feature codes of the pedestrian; and processing the adjacency matrix and the feature representation matrix of each spatial graph using a spatiotemporal convolutional graph neural network to predict the pedestrian crossing action, wherein the pedestrian crossing action is one of the pedestrian crossing in front of the autonomous vehicle or the pedestrian not crossing in front of the autonomous vehicle.

[0021] The present invention also provides a processing system comprising a processor and a memory storing machine-executable instructions, which, when executed by the method, perform any of the above-described computer-implemented methods.

[0022] The present invention also provides a non-transitory computer-readable medium and a storage machine-executable computer program that, when executed by the method, performs any of the computer implementation methods determined above. Attached Figure Description

[0023] The accompanying drawings, which illustrate exemplary embodiments of this application, will now be shown by way of example, in which:

[0024] Figure 1 These are schematic diagrams illustrating exemplary environments for the operation of autonomous vehicles provided in some embodiments of the present invention;

[0025] Figure 2 This is a block diagram of some exemplary components of an autonomous vehicle provided in some embodiments of the present invention;

[0026] Figure 3 This is a block diagram of an exemplary subsystem for predicting pedestrian crossing actions provided in some embodiments of the present invention;

[0027] Figure 4 Examples of observation frames covering a representative spatial frame scene, provided by some embodiments of the present invention, are shown. This representative spatial frame scene can be modeled by a subsystem for predicting pedestrian crossing actions.

[0028] Figure 5 It shows Figure 4 The observation frame has a vertical reference line drawn from the target pedestrian;

[0029] Figure 6 An example of the directional weight values ​​around the target pedestrian is shown;

[0030] Figure 7 It is the second example of an observation frame and has covered spatial map data;

[0031] Figure 8 It shows about Figure 7 The adjacency matrix generated from the observed frames;

[0032] Figure 9 This is a block diagram illustrating an example of another subsystem for predicting pedestrian crossing actions provided in some embodiments of the present invention;

[0033] Figure 10 An example of directional distance that can be used to determine the relative importance value of a location is shown.

[0034] Similar reference numerals can be used to denote similar components in different accompanying drawings. Detailed Implementation

[0035] The technical solutions of the present invention will now be described with reference to the accompanying drawings.

[0036] As used herein, "module" can refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) that can be executed on the hardware processing circuitry. Hardware processing circuitry can include any or some combinations of microprocessors, the core of multi-core microprocessors, microcontrollers, programmable integrated circuits, programmable gate arrays, digital signal processors, or other hardware processing circuitry.

[0037] As used herein, "engine" can refer to hardware processing circuitry, which may include any or a combination of microprocessors, the core of multi-core microprocessors, microcontrollers, programmable integrated circuits, programmable gate arrays, digital signal processors, or other hardware processing circuitry. Alternatively, "engine" can refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) that can be executed on the hardware processing circuitry.

[0038] Some examples of the invention are described in the context of autonomous vehicles. While the examples described herein may refer to automobiles as autonomous vehicles, the teachings of the invention can be implemented in other forms of autonomous or semi-autonomous vehicles, including, for example, trams, subways, trucks, buses, surface and submersible vessels and ships, aircraft, unmanned aerial vehicles (also known as unmanned aerial vehicles, UAVs), warehouse equipment, manufacturing facility equipment, construction equipment, farm equipment, autonomous service robots such as vacuum cleaners and lawnmowers, and other robotic devices. Autonomous vehicles may include both passengerless and passenger-carrying vehicles.

[0039] Examples of the present invention are applicable to applications other than autonomous vehicles. The exemplary methods and systems described herein can be applied to any context in which or it is desirable to accurately predict pedestrian crossing actions in a sensing environment (also known as a sensing scene). A pedestrian is any person or animal in the environment whose state (e.g., location) changes during a period of interest (e.g., more than 10 minutes).

[0040] To help understand the invention, examples of autonomous vehicles in the environment will now be discussed.

[0041] Figure 1 This is a schematic diagram illustrating an exemplary environment 100 in which vehicle 105 operates. Examples of the invention can be implemented in vehicle 105, for example, to achieve autonomous or semi-autonomous driving. Environment 100 includes a communication system 202 that communicates with vehicle 105. Vehicle 105 includes a vehicle control system 115. Vehicle control system 115 is coupled to the drive control system and mechanical system of vehicle 105, as referenced below. Figure 2 Further description. In various examples, the vehicle control system 115 enables the vehicle 105 to operate in one or more of fully automatic, semi-automatic, or fully user-controlled modes.

[0042] Vehicle 105 may include sensors, including multiple environmental sensors 110 shown herein for collecting information about the external environment 100 surrounding vehicle 105 and generating sensor data indicating such information, and multiple vehicle sensors 111 for collecting information about the operating conditions of vehicle 105 and generating vehicle data indicating such information. Different types of environmental sensors 110 may be present to collect different types of information about the environment 100, as discussed further below. In an exemplary embodiment, environmental sensors 110 are mounted and positioned at the front, rear, left, and right sides of vehicle 105 to collect information about the external environment 100 located at the front, rear, left, and right sides of vehicle 105. For a single unit of environmental sensor 110, it may be mounted or otherwise positioned on vehicle 105 to have different overlapping or non-overlapping fields of view (FOV) or coverage areas to capture data about the environment 100 surrounding vehicle 105. Vehicle control system 115 receives sensor data indicating collected information about the external environment 100 of vehicle 105 collected by environmental sensors 110.

[0043] Vehicle sensors 111 provide vehicle data, which is collected and indicates information about the operating conditions of vehicle 105, to vehicle control system 115 in real time or near real time. For example, processing system 115 can use the vehicle data, which indicates information about the operating conditions of vehicle 105 provided by one or more vehicle sensors 111, to determine factors such as the linear velocity of vehicle 105, the angular velocity of vehicle 105, the acceleration of vehicle 105, the engine RPM of vehicle 105, the transmission gears of vehicle 105, and tire grip.

[0044] Processing system 115 includes one or more wireless transceivers 130 (e.g., Figure 1As shown in the diagram, or may be coupled to one or more wireless transceivers 130, which enable the vehicle control system 115 to communicate with the communication system 202. One or more wireless transceivers 130 enable the processing system 115 to communicate with the communication system 202. For example, one or more wireless transceivers 130 may include one or more cellular (RF) transceivers for communicating with multiple different wireless access networks (e.g., cellular networks) or other vehicles in the environment using different wireless data communication protocols and standards. One or more wireless transceivers 130 may communicate with any of multiple fixed transceiver base stations of a wireless wide area network (WAN) 210 (e.g., cellular network) within their geographical coverage area. One or more wireless transceivers 130 may transmit and receive signals via the wireless WAN 210. One or more wireless transceivers 130 may include multi-band cellular transceivers supporting multiple radio frequency bands. The processing system 115 may use the wireless WAN 210 to access the server 240 (e.g., a driver assistance server) via one or more communication networks 220 (e.g., the Internet). Server 240 can be implemented as one or more server modules in a data center and is typically located behind firewall 230. Server 240 can connect to network resources 250, such as supplementary data sources that can provide information used by the vehicle control system 115.

[0045] One or more wireless transceivers 130 may also include a wireless local area network (WLAN) transceiver for communicating with a WLAN (not shown) via a WLAN access point (AP). The WLAN may include devices conforming to the IEEE 802.11x standard (sometimes referred to as...). A Wi-Fi wireless network using one or more other communication protocols. One or more wireless transceivers 130 may also include short-range wireless transceivers, such as... A transceiver for communicating with mobile computing devices, such as smartphones or tablets. One or more wireless transceivers 130 may also include other short-range wireless transceivers, including but not limited to near field communication (NFC), IEEE 802.15.3a (also known as ultra-wideband (UWB)), Z-Wave, ZigBee, ANT / ANT+, or infrared (such as Infrared Data Association (IrDA) communications).

[0046] The communication system 100 also includes a satellite network 260 comprising multiple satellites. The processing system 115 can determine its location using signals received from multiple satellites in the satellite network 260. The satellite network 260 typically includes multiple satellites that are part of at least one global navigation satellite system (GNSS). At least one GNSS provides automatic geospatial positioning with global coverage. For example, the satellite network 260 may be a collection of GNSS satellites. Exemplary GNSS include the US NAVSTAR global positioning system (GPS) or the Russian global navigation satellite system (GLONASS). Other deployed or under-development satellite navigation systems include the European Union's Galileo positioning system, China's BeiDou navigation satellite system (BDS), India's regional satellite navigation system, and Japan's satellite navigation system.

[0047] The electromechanical system 190 receives control signals from the drive control system 150 to operate the electromechanical components in the vehicle 105. The electromechanical system 190 affects the physical operation of the vehicle 105. The electromechanical system 190 includes an engine 192, a transmission 194, and wheels 196. The engine 192 can be a gasoline-powered engine, a battery-powered engine, a hybrid engine, etc. Other components may be included in the mechanical system 190, including turn signals, brake lights, a fan, and windows.

[0048] Figure 2 Selected components of a vehicle 105 provided with some examples described herein are shown. The vehicle 105 includes a vehicle control system 115, which is coupled to a drive control system 150 and an electromechanical system 190. The vehicle control system 115 is also coupled to receive data from environmental sensors 110 and vehicle sensors 111.

[0049] The environmental sensor 110 may include, for example, one or more camera units 112, one or more light detection and ranging (LIDAR) units 114, and one or more synthetic aperture radar (SAR) units 116, as well as other possibilities. Each type of sensor unit 112, 114, 116 can collect different information about the external environment 100 of the vehicle 105 and can provide sensor data to the vehicle control system 115 in different formats. For example, camera unit 112 can provide camera data representing digital images, LIDAR unit 114 can provide two-dimensional or three-dimensional point clouds, and SAR unit can provide radar data representing radar images.

[0050] Vehicle sensor 111 may include, for example, an inertial measurement unit (IMU) 118 that senses the specific force and angular rate of vehicle 105 and provides data on vehicle orientation based on the sensed specific force and angular rate. Vehicle sensor 111 may also include an electronic compass 119 and other vehicle sensors 120, such as a speedometer, tachometer, wheel traction sensors, transmission gear sensors, throttle and brake position sensors, and steering angle sensors.

[0051] The vehicle control system 115 can also use signals received from the satellite network 260 via the satellite receiver 132 to collect information about the location of the vehicle 105 and generate positioning data representing the location of the vehicle 105.

[0052] Vehicle 105 also includes various structural elements such as frames, doors, panels, seats, windows, mirrors, etc., which are known in the art but have been omitted from this invention to avoid obscuring its teachings. Vehicle control system 115 includes a processor system 102 coupled to multiple components via a communication bus (not shown), which provides communication paths between the components and the processor system 102. The processor system 102 is coupled to a drive control system 150, random access memory (RAM) 122, read-only memory (ROM) 124, persistent (non-volatile) memory 126 such as erasable programmable read-only memory (EPROM), one or more wireless transceivers 130, a satellite receiver 132, and one or more input / output (I / O) devices 134 (e.g., touchscreens, speakers, microphones, displays, mechanical buttons, etc.). The processor system 102 may include one or more processing units, including one or more central processing units (CPUs), one or more graphics processing units (GPUs), one or more tensor processing units (TPUs), and other processing units.

[0053] The drive control system 150 provides control signals to the electromechanical system 190 to achieve physical control of the vehicle 105. For example, when in fully automatic or semi-automatic driving mode, the drive control system 150 receives planned actions (discussed further below) from the vehicle control system 115 and uses the steering unit 152, braking unit 154, and throttle (or acceleration) unit 156 to convert the planned actions into control signals. Each unit 152, 154, 156 can be implemented as one or more software modules or one or more control blocks within the drive control system 150. The drive control system 150 may include other components for controlling other aspects of the vehicle 105, including controlling turn signals and brake lights, etc.

[0054] The electromechanical system 190 receives control signals from the drive control system 150 to operate the electromechanical components in the vehicle 105. The electromechanical system 190 affects the physical operation of the vehicle 105. The electromechanical system 190 includes an engine 192, a transmission 194, and wheels 196. The engine 192 can be a gasoline-powered engine, a battery-powered engine, a hybrid engine, etc. Other components may be included in the mechanical system 190, including turn signals, brake lights, a fan, and windows.

[0055] The memory 126 of the processing system 115 stores software instructions executable by the processor system 102. These software instructions can be executed by the processor system 102 to implement one or more software systems, software subsystems, and software modules. Generally, it should be understood that the software systems, software subsystems, and software modules disclosed herein can be implemented as a set of instructions stored in the memory 126. For example, the memory 126 may include executable instructions for implementing the operating system 160, the perception system 200, and the planning system 300 (also referred to as a path planning system). The perception system 200 and the planning system 300 may be part of the Adaptive Devices (ADS) of the vehicle 105 (not shown). The planning system 300 may be a machine learning-based system that generates planned paths (which may include planned sub-paths and planned behaviors) to be executed by the vehicle 105. The planning system 300 includes a task planning subsystem 305, a behavior planning subsystem 310, a motion planning subsystem 315, and an object behavior prediction subsystem 320. The object behavior prediction subsystem 320 also includes a pedestrian action prediction subsystem 400. Details of the pedestrian crossing action subsystem 400 will be provided below.

[0056] The planning and decision-making of the planning system 300 can be dynamic and repeated as the environment changes. The changes in the environment can be due to the movement of vehicle 105 (e.g., vehicle 105 approaching a newly detected obstacle) and due to the dynamic nature of the environment (e.g., moving pedestrians and other moving vehicles).

[0057] Planning subsystems 305, 310, and 315 perform planning at different levels of detail. Task-level planning performed by task planning subsystem 305 is considered a higher (or more global) planning level, motion-level planning performed by motion planning subsystem 315 is considered a lower (or more local) planning level, and behavior-level planning performed by behavior planning subsystem 310 is considered a planning level between task and motion levels. Typically, the output from a higher-level planning subsystem can constitute at least a portion of the input to a lower-level planning subsystem.

[0058] The planning (more simply referred to as task planning) of the task planning subsystem 305 involves planning the path of vehicle 105 at a high or global level, such as planning a route from a starting point (e.g., home address) to a final destination (e.g., work address). The behavior planning subsystem 310 can receive planned routes from the task planning subsystem 305. The behavior planning subsystem 310 involves controlling the behavior of vehicle 105 on a more local and short-term basis than the task planning subsystem 305. The behavior planning subsystem 310 can generate behavioral decisions based on certain rules (e.g., traffic rules, such as speed limits or signs) or guidance (e.g., guidance for smooth and efficient driving, such as taking the faster lane if possible). These behavioral decisions can be provided as part of the input to the motion planning subsystem 315. The motion planning subsystem 315 involves controlling the motion of vehicle 105 based on the immediate environment 100 of vehicle 105. The motion planning subsystem 315 generates planned vehicle motion to ensure the safety of vehicle occupants and other objects in the environment (e.g., pedestrians, other vehicles, cyclists, etc.). Since the environment can be highly dynamic (e.g., pedestrians and other vehicles are moving), the motion planning subsystem 315 should perform motion planning that takes into account expected (or predicted) changes in the environment 100.

[0059] Sensor data received from environmental sensor 110 and vehicle data received from vehicle control sensor 111 (and optionally positioning data collected from satellite network 260) can be used by perception system 200 to generate processed data (e.g., observation frames, feature vectors, occupancy grid map (OGM), object classification, and bounding boxes, etc.) representing features of environment 100 at each time step. Perception system 200 may include one or more machine learning-based systems (e.g., trained neural networks) that generate processed data representing features of environment 100 at each time step.

[0060] The perception system 200 may include any number of independent or interconnected systems or functions, and may include, for example, rule-based systems, machine learning-based systems, and combinations thereof. Machine learning-based systems may be implemented using neural networks, such as any type of DNN (including CNN or RNN), Long Short-Term Memory networks, etc. In some examples, the perception system 200 may include: a fusion subsystem for fusing sensor data and vehicle data from multiple environmental sensors 110 and vehicle sensors 111 to generate fused data; a classification subsystem for processing the sensor data or fused data to detect and identify objects in the environment 100 (e.g., detecting and identifying stationary obstacles, pedestrians or other vehicles, lanes and lane boundaries, and traffic lights / signs, etc.); and a localization and mapping subsystem 406 for building or updating a map of the environment 100 and estimating the position of vehicle 105 in the map. In an exemplary embodiment, the localization and mapping subsystem of the perception system 200 is used to generate a bird's-eye view (BEV) map (also known as a top-down view map) in real time, representing the scene near vehicle 105 as observed from a two-dimensional bird's-eye view. In an exemplary embodiment, the BEV map can be generated based on a combination of a rasterized high-definition (HD) top-down map of the surrounding environment and the results of dynamic object detection performed by the classification subsystem of the perception system 200. In at least some examples, information from sensors outside the vehicle 105 (e.g., pole-mounted cameras) can be received by the wireless transceiver 130 of the processing system 115 and incorporated into the BEV map. In an exemplary embodiment, a corresponding 2D BEV map of the environment 100 is generated for each of a series of time steps. Each 2D BEV map can correspond to an observation frame of the environment.

[0061] The memory 126 may also store instructions thereon for implementing other software systems, subsystems and modules, such as navigation systems, climate control systems, media player systems, telephone systems and / or messaging systems.

[0062] Figure 3 This is a block diagram illustrating exemplary details of a pedestrian action prediction subsystem 400. The pedestrian action prediction subsystem 400 may be a subsystem of the object behavior prediction subsystem 320 (which in turn may be a subsystem of the planning system 300). However, it should be understood that the pedestrian action prediction subsystem 400 may also be used outside of the object behavior prediction subsystem 320, outside of the planning system 300, and / or outside of the ADS (not shown) of the autonomous vehicle 105. For example, the pedestrian action prediction subsystem 400 can be applied to any computing system (e.g., assistive robots, intelligent traffic management, etc.) where predicting pedestrian actions (i.e., pedestrian movements) would be useful.

[0063] The pedestrian motion prediction subsystem 400 includes a graph modeler 402 and a pedestrian motion predictor 404. The graph modeler 402 includes a scene context modeling module 408, a feature encoder 410, and a scene road user clustering module 412. The graph modeler 402 receives processed data from the perception system 200 of the vehicle 105, representing features of the environment 100 at each time step within a predetermined time period. In some examples, the processed data includes observation frames 401 (i.e., observation scenes) for each corresponding time step. In some examples, each observation frame 401 may take the form of a 2D bird's-eye view (BEV) map of the environment 100. As described above, the time-step 2D BEV map can be generated by the localization and mapping subsystem 406 of the perception system 200. In some examples, some or all of the functionality of the localization and mapping subsystem 406 can be incorporated into a preprocessor module, which can be added to the front end of the graph modeler 402. Each observation frame 401 includes metadata and spatial information classifying the corresponding object represented in the observation frame (e.g., pedestrian, vehicle, bicycle). In this respect, observation frame 401 includes spatial information about the target pedestrian (i.e., the pedestrian whose actions are to be predicted) and spatial information about other dynamic objects surrounding the target pedestrian in environment 100. In this invention, dynamic objects that can potentially interact with pedestrians, besides the target pedestrian, are referred to as road users and may include motor vehicles, self-propelled vehicles such as bicycles, pedestrians, animals, etc. The spatial information includes information that allows the relative positions of the target pedestrian and road users to be determined from a single observation frame 401, and information that determines the movement directions of the target pedestrian and road users based on the time series of observation frames 401.

[0064] Scene Context Modeling Module

[0065] The scene context modeling module 408 receives observation frames 401; for each corresponding observation frame 401, it determines the position of the target pedestrian and the position of each road user; it generates a spatial graph G = (V, E) for the corresponding observation frame 401 (where V is the set of nodes v and E is the set of connecting edges e), and generates a corresponding adjacency matrix A1 for the spatial graph G of the corresponding observation frame 401. The nodes V in the spatial graph G generated for each corresponding frame include the center node (v) representing the target pedestrian. p ) and multiple other nodes (v u ), where each of the other nodes (v ru () represents one of the road users. The corresponding connecting edge e in the spatial graph of each observation frame represents the center node v. p With each other node v ruThe direct connections between them. The adjacency matrix A1 consists of multiple elements, where each element of the adjacency matrix A1 includes the direct connections between the target pedestrian (assigned by the center node v of the spatial graph G). p (represented) and one of the road users (by other nodes v in the spatial graph) ru The weights of direct connections (e.g., edges e) between two points (represented in one of the representations).

[0066] Figure 4 An illustrative example of observation frame 401 and its corresponding spatial graph G is shown. Figure 4 In this context, observation frame 401 corresponds to a 2D BEV map, which includes detected objects representing three vehicles (road users) and a target pedestrian located on or near the road. As an example, observation frame 401 may have already been generated by the perception system of vehicle 105. Scene context modeling module 408 generates a corresponding spatial map G (overlaid for illustrative purposes) based on the spatial information represented in observation frame 401. Figure 4 The observation frame 401 in the image (and the adjacency matrix A1) are used. Specifically, the target pedestrian is represented as the center node v. P And the corresponding vehicle is represented as a road user node v. ru 1. v ru 2. v ru 3. Connecting edges e1, e2, and e3 represent the central target pedestrian node v. P With the corresponding road user node v ru 1. v ru 2. v ru The direct connections between each of the nodes in 3. The corresponding lengths of the connecting edges e1, e2, and e3 represent the real-world distances between the target pedestrian and each corresponding road user, and these distances between nodes are determined by the scene context modeling module 408 using one of the multiple factors (e.g., a1, a2, a3) in the adjacency matrix A1 to determine the corresponding multi-factor importance weights.

[0067] The multi-factor importance weights (e.g., a1, a2, a3) assigned by the scene context modeling module 408 to each direct connection (e.g., connecting edges e1, e2, e3) in the spatial graph G represent the relative importance of each road user to the target pedestrian. In the example, the scene context modeling module 408 assigns a higher weight to a... i Direct connections (i.e., connection edges e) assigned to road users who are both close to the target pedestrian and in a relatively important position relative to the target pedestrian in observation frame 401. i In some embodiments, the scene context modeling module 408 assigns a scene context modeling module 408 to each directly connected element in the spatial graph G. i Multifactor importance weights a iThis is based on: (i) the distance between the road user and the target pedestrian, and (ii) the relative position between the road user and the target pedestrian. In some embodiments, the scene context modeling module 408 assigns a scene context modeling module 408 to each directly connected element in the spatial graph G. i Multifactor importance weights a i The following determinations are made.

[0068] For the target pedestrian, determine the distance from the target pedestrian in observation frame 401 to each road user surrounding the target pedestrian. A road user can be another pedestrian, a bicycle, or a vehicle. The distance from the target pedestrian to the road users is represented by the vector D = {d1, d2, ..., d...}. i ...,dN} represents the number of road users, where N is the number of road users, and the i-th element represents road user i (represented as road user node v in the spatial graph G). ru i Relative to the target pedestrian (represented as the target pedestrian node v in the spatial graph G). p The distance of the road user to the target pedestrian is then normalized by the value dmax, which is the furthest distance considered when predicting the target pedestrian's movement (e.g., represented by the longest connecting edge e). The farther the road user is from the target pedestrian, the closer their corresponding normalized distance is to 1. In some embodiments, (1–D) can be used as the first importance weight for each road user around the target pedestrian (hereinafter referred to as the "distance importance weight").

[0069] When the multi-factor importance weights a representing the importance of each road user i are applied... i When assigning a road user to a target pedestrian, the scene context modeling module 408 also models the relative directional position of each road user around the target pedestrian. Based on the relative position of the respective road user relative to the target pedestrian, the scene context modeling module 408 assigns a second importance weight (i.e., the relative position importance weight b). i ) is assigned to each corresponding road user i around the target pedestrian.

[0070] refer to Figure 5 In some embodiments, based on the connection edge e between road user i and target pedestrian i The angle position will be weighted by the relative position importance b. i A reference line L is assigned, which is drawn from the position of the target pedestrian along a direction perpendicular to the road (e.g., perpendicular to the legal driving direction of the road user in the lane closest to the target pedestrian). For each of the N road users around the target pedestrian (i.e., the target pedestrian is the center or origin of the reference line L), the connecting edge e perpendicular to the reference line L is measured (between the target pedestrian and each corresponding road user i). iThe counterclockwise angle θi between the reference line L and the road user i. For each road user i, the counterclockwise angle θi measured from the reference line L will be between 0 and 360 degrees. In one example, the scene context modeling module 408 assigns a weight b to the relative position importance of road user i relative to the target pedestrian. i Defined as:

[0071]

[0072] Scene context modeling module 408 generates a relative position importance weight vector B = {b1, b2, ..., bi, ..., bN}, where the i-th element of vector B (e.g., b1, b2, ..., bi, ..., bN) is the relative position importance weight. i This indicates the relative positional importance of road user i relative to the target pedestrian. It should be noted that Equation 1 assigns a weight b to the relative positional importance. i The variable is determined as a continuous variable assigned to road user i based on the relative directional position of road user i. Figure 6 The relative position importance weight b is calculated relative to the vertical reference line L at angles θi = 90 degrees, 180 degrees, 270 degrees, and 0 degrees. i Wherein, the target pedestrian 606 is at the origin of reference line L. When pedestrian 604 is facing the road, the following relative positional importance weights will be assigned to the objects in front of, behind, to the left and to the right of the target pedestrian: front = 1 / 2; behind = 1 / 2; left = +1; right = 0.

[0073] As described above, the scene context modeling module 408 assigns less relative positional importance weight to road users positioned to the right of the target pedestrian and more relative positional importance weight to road users positioned to the left of the target pedestrian. More specifically, road users positioned to the right of reference line L are assigned less relative positional importance weight than road users to the left of the target pedestrian. For example, this relative weighting is applicable to areas where driving on the right side of the road is legally mandated.

[0074] In some other embodiments, discrete values ​​can be assigned instead of continuous values ​​to represent relative positional importance, for example, in Figure 6 In this case, road users at relative positions relative to the vertical reference line L at a ratio of 0 degrees <= θi <= 180 degrees can be considered important, and a relative position importance weight b is assigned. i =1, and road users at relative positions that are 180 degrees < θi < 360 degrees relative to the vertical reference line L can be considered unimportant, and a relative position importance weight b is assigned. i =0. However, using continuous weights is more accurate.

[0075] The above method for calculating relative position importance weights is illustrative; any formula that assigns importance weights reflecting expected road user behavior to surrounding road users can be used for weight allocation in vector B.

[0076] When the scene context modeling module 408 has already calculated both the distance importance vector D and the relative position importance weight vector B, the final multi-factor importance weight a is... i Assigned to each road user as follows:

[0077] A = B.(1–D)

[0078] Here, . denotes element-wise multiplication of vectors. Vector A = {a1, a2, ..., a...} i The i-th element of the multi-factor importance weights aN is relative to the target pedestrian. i If the value is close to 0, then road user i is either far away from the target pedestrian or to the right of the target pedestrian.

[0079] Accordingly, the spatial graph G is a star graph centered on the target pedestrian. The central node of the star graph represents the target pedestrian, and each other node in the star graph represents a road user. Direct connections in the star graph connect each road user to the target pedestrian. The relative distance and orientation of the user node with respect to the target pedestrian are incorporated into the corresponding edge weights, as shown in the adjacency matrix A.

[0080] Feature encoder

[0081] The feature encoder 410 receives spatial information from the preprocessing module 404 for each observation frame 401, generates spatiotemporal information about the target pedestrian based on the spatial information about the target pedestrian in the set of observation frames 401, and generates spatiotemporal information about each corresponding road user based on the spatial information about the corresponding road user in the set of observation frames. The feature encoder 410 also generates feature codes x for the target pedestrian. p The feature encoding x p The spatiotemporal information generated for the target pedestrian is encoded. The feature encoding of the target pedestrian, x, is... p It can be in vector form. Spatiotemporal information about the target pedestrian is encoded as feature code x. p The corresponding features of the corresponding elements may include one or more of the following: the location of the target pedestrian, the size of the target pedestrian, the speed of the target pedestrian, the direction of movement of the road user, and the distance from the target pedestrian to the boundary of the road on which the vehicle 105 is traveling.

[0082] The feature encoder 410 also generates feature codes x for each road user i. rui The feature encoding x for each road user i ru i The generated spatiotemporal information about road users is encoded. The feature encoding x for each road user i is... ru i It can be in vector form. Spatiotemporal information about each corresponding road user i is encoded as a feature code x. ru i The corresponding features of the corresponding elements in the vector can include one or more of the following: the position of the corresponding road user relative to the target pedestrian, the size and / or type of the corresponding road user (e.g., car, bus, bicycle, other pedestrian), the speed of the corresponding road user, the direction of movement of the corresponding road user, the distance from the corresponding road user to the boundary of the road on which vehicle 105 is traveling, and the speed of the road user. Therefore, the feature encoder 410 generates a feature vector {x}. p ,x ru 1,...,x ru N The feature encoder 410 can use historical information from previous observation frames 401 to determine the feature matrix X for the current observation frame.

[0083] Scene Road User Clustering Module

[0084] In the example where the graph modeler 402 includes a scene road user clustering module 412, the scene road user clustering module 412 is used to identify road users with similar behaviors and merge their similarities into a spatial graph G generated for each observation frame 401. The scene road user clustering model 412 clusters road users with similar behaviors into the same group. The scene road user clustering model 412 uses some criteria to determine whether the behaviors of some road users are similar. In order to cluster objects into groups, the scene road user clustering module 412 can perform clustering operations according to clustering criteria or models and based on information derived from the current time step and historical observation frames 401, for example as follows:

[0085] (1) Based on the type and movement of road users, they are divided into different groups: stationary pedestrians, moving pedestrians, stationary vehicles, moving vehicles, stationary bicycles, and moving bicycles.

[0086] (2) Based on the orientation of non-pedestrian road users, the non-pedestrian road users are further clustered into different groups (e.g., motor vehicles and bicycles). The specific method is: (a) determine the orientation vector based on the annotation; (b) use the KMEAN algorithm to divide the orientation vector into two subgroups (e.g., clusters), one subgroup for each driving direction; (3) based on the orientation vector, divide the subgroups (e.g., clusters) in step (2)(b) into another subgroup (e.g., subclusters).

[0087] (3) Based on the orientation of pedestrian road users, further cluster them into subgroups (e.g., clusters). If pedestrians are stationary, there is no need to further divide them into more subgroups (e.g., subclusters). If pedestrians are moving, pedestrian clusters are divided from step (3) based on the pedestrians' orientation and / or movement. If any two pedestrians are facing the same direction, the two pedestrians are grouped into the same cluster. If any two pedestrians are facing different directions and moving towards each other, the two pedestrians are grouped into the same cluster. If any two pedestrians are facing opposite directions and moving away from each other, the two pedestrians are divided into two subgroups. In some examples, determining whether two pedestrians are moving in the same direction can be performed by determining the angle between the moving / standing directions of the two pedestrians; if the angle is less than 90 degrees, the two pedestrians are determined to be moving in the same direction. Otherwise, the two pedestrians are determined to be moving in different directions.

[0088] Clustering is complete when the minimum possible clusters are obtained according to the predetermined clustering criteria or model.

[0089] After obtaining the clusters, the spatial graph G for each observation frame is modified to include edge connections between r nodes representing road users and included in the same group (cluster). The final spatial graph G for each observation frame will consist of a star graph, where the central node represents the target pedestrian performing pedestrian action prediction, and the other nodes representing road users within the cluster are fully interconnected. The fully connected other nodes are of the same type (motor vehicles, pedestrians, bicycles, etc.), and these nodes have been identified as having similar behaviors.

[0090] For example, in Figure 7 In this context, road users within the same ellipsoid belong to the same cluster (e.g., motor vehicle type road users 402_1 and 402_2 in ellipsoid 702_1 are in the first common cluster; motor vehicle type road users 402_3 to 402_5 in ellipsoid 702_2 are in the second common cluster; and bicycle type road users 402_6 and 402_7 in ellipsoid 702_3 are in the third common cluster). Each corresponding cluster contains a road user node v. ru Connect the corresponding edges to each element, thus adding another set of edges to the spatial graph G. Figure 7In the diagram, the solid lines connecting pedestrian 404 and the corresponding road users 402_1 to 407_7 represent the original connection edges determined by the scene context modeling module 408. The dashed lines connecting the corresponding road users within each cluster (i.e., within each ellipsoid 702_1, 702_3, and 702_3) represent the intra-cluster connection edges added in the scene road clustering module 412.

[0091] In an exemplary embodiment, each intra-cluster connecting edge has a weight of 1. The adjacency matrix A1 can be extended to include elements representing intra-cluster connecting edges, thereby producing a comprehensive adjacency matrix A2. Figure 8 The text shows the corresponding... Figure 7 An example of an adjacency matrix A2, where v p This represents the center node of the target pedestrian 404, v ru 1 to v ru 7 represents the road user nodes 402_1 to 402_7, respectively.

[0092] Advantageously, the scene road user clustering module 412 generates a spatial map of the observation frames, which includes a better understanding of the observation frames (i.e., the traffic scene). The spatial map of each observation frame is provided as a spatiotemporal map to the pedestrian action predictor 404, which, as described below, predicts the future pedestrian actions of a target pedestrian.

[0093] Pedestrian motion predictor

[0094] The pedestrian action predictor 404 receives a spatial graph of the observed frame set and processes the spatial graph using a spatiotemporal graph convolutional neural network (GCNN) to predict the pedestrian actions of the target pedestrian. Specifically, the pedestrian action predictor 404 receives a feature matrix X and an adjacency matrix A2 as input. The feature matrix X includes the corresponding feature vector encodings of the target pedestrian and each of the N road users. The adjacency matrix A2 identifies the relative importance weights a1 to aN between the target pedestrian and each of the N road users, and also identifies the corresponding clusters of road users with similar behaviors. The GCNN generates probability values ​​corresponding to pedestrian actions that the pedestrian will cross the road and probability values ​​corresponding to pedestrian actions that the pedestrian will not cross the road, and outputs a binary value representing the predicted pedestrian action (whether or not to cross the road). For example, if the generated probability value corresponding to a pedestrian action that the pedestrian will cross the road is higher than the probability value corresponding to a pedestrian action that the pedestrian will not cross the road, the pedestrian action predictor 404 outputs an indication (i.e., a label) that the pedestrian action is that the pedestrian will cross the road. Similarly, for example, if the probability value of generating a pedestrian action corresponding to a pedestrian crossing is lower than the probability value of generating a pedestrian action corresponding to a pedestrian not crossing, then the pedestrian action predictor 404 outputs an indication (i.e., a label) that the pedestrian action is a pedestrian not crossing.

[0095] Based on the generated predictions, the planning system 300 of the vehicle 105 can determine the appropriate actions that one or more systems of the vehicle 105 should take. For example, the planning system 300 can issue a command to the drive control system 150 to cause the braking unit 154 to decelerate the vehicle 105. Other actions that may be indicated include: changing the steering angle; increasing or decreasing the throttle; sounding the horn; and activating interior warnings (audible and / or visual and / or tactile alarms) in the vehicle cabin.

[0096] Therefore, in this example, the pedestrian action subsystem 400 applies a graph-based interaction modeling method that takes a bird's-eye view of road users as input. Importance weights are assigned to different road users based on their distance and relative position to the pedestrian.

[0097] An example of another pedestrian motion prediction subsystem 900 will now be described according to another exemplary aspect of the invention.

[0098] In the pedestrian motion prediction subsystem 900, pedestrian motion prediction is treated as an optimization problem. The objective is to determine the location L of the target pedestrian, vehicle 105, and nearby road users. o ={l1, l2, ..., l T} and directional O o ={o1, o2, ..., o T} and the traffic direction R of the road o ={r1, r2, ..., r T Given a global bird's-eye view of the observation sequence, estimate the probability distribution of the crossing action of the i-th pedestrian at some future time t+k. in,

[0099] Similar to the action prediction subsystem 400, the action prediction subsystem 900 includes a graph modeler 902 and a pedestrian action predictor 916. The graph modeler 902 includes: (i) a road user clustering module 912 for grouping road users near the target pedestrian using their speed, location, and orientation; and (ii) a feature encoder module 910 for forming feature vectors for each observation (e.g., road users included in observation frame 401) and the target pedestrian. These vectors serve as node representations of road users and the target pedestrian in the spatial graph G. The graph modeler 902 also includes (iii) an interaction modeling module 914 for calculating the relative importance of road users to the target pedestrian, which is used in conjunction with the identified clusters to determine the weights of connecting edges in the spatial graph G. The pedestrian action predictor 916 uses the graph representation of the spatial graph G and some dynamic information about the target pedestrian and vehicles to predict the probability of the target pedestrian's future crossing actions.

[0100] Road User Clustering Module 912

[0101] In the example, the road user clustering module 912 is used to group road users based on similar behaviors. In this regard, the road user clustering module 912 performs frame-by-frame clustering of road users around a target pedestrian based on the static and dynamic characteristics of the road users surrounding the target pedestrian, including object type and speed, location, and orientation, as follows:

[0102] Object type.

[0103] First, road users are categorized into several types, such as three categories—pedestrians, vehicles, and bicycles. Then, for the observation sequence F... o ={f1,f2,...,f T Each observation frame 401 in} is first based on velocity S o ={s1,s2,...s} T}, then based on the bird's-eye view location L o Finally, based on directional O o By using a hierarchical approach, road users are further clustered into each category, thus obtaining a cluster set for each observation frame, C o ={c1,c2,...,c T}

[0104] Object speed.

[0105] Within each category, the speed of each road user is calculated between the previous and current frames to determine whether they are moving or stationary. For the i-th object in frame t,

[0106]

[0107] Where t = 2...T, the frame rate is the frequency of observed frame 401, in Hz. At t = 1, s1 (i) Set to 0.

[0108] Empirically determined speed thresholds (e.g., 0.2 m / s, 2 m / s, and 2 m / s) are used to classify each object category into a moving subgroup or a stationary subgroup. If s t (i) If the threshold is greater than or equal to the threshold, then road user i is considered to be in the observation frame f. t Moving. Therefore, using road user type and speed, all road users in observation frame 401 are divided into several groups, such as six groups: stationary pedestrians, moving pedestrians, stationary vehicles, moving vehicles, stationary bicycles, and moving bicycles.

[0109] Location.

[0110] Next, coarse clusters are generated for each group of road users based on the distances between them in the bird's-eye view. For example, this can be done using the density-based spatial clustering of applications with noise (DBSCAN) algorithm [see M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, "Adensity-based algorithm for discovering clusters in large spatial databases with noise," International Conference on Knowledge Discovery and Data Mining, 1996]. DBSCAN requires two parameters—the maximum distance between two points considered in the same cluster and MinPts used to determine the noise level in the cluster. MinPts can be set to 1 so that no object (e.g., road users) is identified as noise. In an illustrative example, the maximum distance can be empirically set to 1.5 meters, 10 meters, and 5 meters for pedestrians, vehicles, and bicycles, respectively.

[0111] Orientation.

[0112] These distance-based clusters can be further subdivided based on the orientation information of road users. In an illustrative example, no further clustering is performed for stationary pedestrians because the orientation of pedestrians in the stationary group is highly variable. For moving pedestrians, clustering is based on two factors—(1) whether two given pedestrians are facing the same direction or different directions, and (2) whether they are moving toward or away from each other. In this regard, two pedestrians can be considered as being in the same location in frame f if the angle between their orientation vectors γ satisfies the following equation. t Objects i and j facing opposite directions:

[0113]

[0114] in, and These are the orientation vectors of objects i and j, respectively. Otherwise, pedestrians are considered to be facing the same direction.

[0115] If the distance between two pedestrians is measured from the observation frame f t–1 Increase to f t If they are moving away from each other, then it is assumed they are moving away from each other. In other words, if Pedestrians are considered to be moving toward each other. If two pedestrians are: (1) facing the same direction, or (2) facing opposite directions and moving toward each other, they will remain in the same distance cluster. If two pedestrians in a distance cluster are facing opposite directions and moving away from each other, they will be divided into different clusters.

[0116] For vehicle and bicycle clusters, the K-means algorithm can be used, for example, to cluster them based on their normalized orientation vectors, dividing road users into two groups. The purpose of this is to prevent vehicles and bicycles traveling in different directions from being grouped into the same cluster.

[0117] As described below, the interaction modeling module 914 is used in the observation frame f t The cluster set found, i.e., c t ={c (1) t ,c (2) t,...,c (NC,t)}, where NC,t is the observation frame f t The total number of clusters used to determine the weights of graph edges.

[0118] Feature encoder module 910

[0119] In the observation frame f t The target pedestrians and each road user in the traffic scene represented in the figure are represented in the figure G as a given observation frame f. t The corresponding object node. In one example, observe frame f. t The node representation of the i-th object in the vector is represented as the object node vector v. (i) t For example, the vector v of an object (e.g., a road user or a target pedestrian). (i) t It can include 35 elements, which can be used to specify corresponding feature values ​​that include information such as object type, location, movement, and size. In this respect, each road user and target pedestrian is represented by a set of feature values ​​that includes information beyond the bird's-eye view coordinates. For example, for each object, the object node vector v (i) t The set of seven elements out of the 35 elements can be used to specify seven feature values ​​directly belonging to the i-th object. Of these seven feature values, the first two indicate whether the object is stationary [1,0] or moving [1,0]. Including this information allows the spatial graph convolution operation to encode information about the ratio of the importance of stationary road users to moving road users relative to the target pedestrian. The third and fourth feature values ​​represent the object's position relative to the target pedestrian. For the target pedestrian, these values ​​will be [0,0]. Otherwise, for road user i, these values ​​are... A threshold distance (e.g., d) is set based on experience.thresh =20m) can be defined around the target pedestrian, and the coordinates [x,y] can be used Adjusted to a range of 0 to 1, where d is the distance from the road user to the target pedestrian along the global x or y axis. The fifth eigenvalue is the road user's speed s. t(i) As defined in the object velocity equation above. The sixth and seventh feature values ​​are the object's length and width, respectively. In some examples, the maximum values ​​of velocity, length, and width for each object type (e.g., vehicle, bicycle, pedestrian) are determined during training of the prediction action subsystem 900, and these values ​​are rescaled to a range of 0 to 1 in the training set. In this respect, a scaling factor can be learned to scale the feature values ​​to a similar range during system deployment. After obtaining the seven feature values ​​of the object, the object node vector v is populated based on the object's type (vehicle, target pedestrian, other pedestrian, other vehicle, and bicycle). (i) t Specific elements.

[0120] In an exemplary embodiment, the object node vector v (i) t Different parts of the element's position are assigned to different types of objects. For example, if a node represents a target pedestrian, then the node vector v... (i) t The first seven element slots (e.g., the first part consisting of element positions 1 to 7) are each filled with seven eigenvalues. For the node vector v representing vehicle 105... (i) t The next seven element slots (e.g., the second part consisting of element positions 8 to 14) are used for feature values. For other pedestrians, other vehicles, and bicycles, the node vector v (i) t The third, fourth, and fifth elements are used for the seven feature values, respectively. The purpose of separating feature values ​​for different object types is to help the model identify the types of objects around the target pedestrian.

[0121] For N unique objects in the observation frame sequence, for each observation frame, the feature vectors are stacked into a matrix of size N×Q, where Q is the node vector v. (i) t The length of the element in the time slot. (For example, in the example shown, Q = 35), thus obtaining the feature matrix set V. o ={V1, V2, ..., V T}. T represents the total number of time steps (e.g., observation frames) that will be treated as a set by the pedestrian action predictor 916. When preparing the spatiotemporal convolution of the pedestrian action predictor 916, the sequence of feature matrices is stacked into a 3D tensor of size N×Q×T.

[0122] Interactive Modeling Module 914

[0123] Interactive modeling module 914 is used for each observation frame f in the observation sequence of the target pedestrian, based in part on the clustering generated by road user clustering module 912. t Construct the graph structure (graph topology of interconnected object nodes) of the spatial graph G to form a symmetric adjacency matrix A. The symmetric adjacency matrix A and the node feature vectors V generated by the feature encoder module 910 are then used to construct the graph structure. o Combined, they form the observation frame f t Spatial diagram G.

[0124] In this regard, the interaction modeling module 914 constructs a graph topology, in which the target pedestrian node (e.g., the node representing the target pedestrian) is connected to the observation frame f via edges. t The diagram shows the connection of each node for each road user appearing in the scene. Then, based on the observation frame f... t generated clusters Each road user node is connected to every other road user node in its own cluster via edges, forming a fully connected cluster subgraph.

[0125] In the observation frame f t The symmetric adjacency matrix A at point t Connecting these nodes or representing graph edges as located in the adjacency matrix A t The corresponding value at the element position in the table corresponds to the intersection of the rows and columns of different object nodes at these positions (i.e., Figure 8 (The same structure shown in matrix A2 of the previous exemplary embodiment).

[0126] Symmetric adjacency matrix A t The values ​​of the connecting edges between the target pedestrian node and each corresponding road user node are determined by two matrices: Bt is a matrix encoding the relative positional importance of road users to the target pedestrian, and Dt is a matrix representing the distance between the target pedestrian and each object. For consistency, and to describe the pedestrian action subsystem, the target pedestrian node will be assigned to an adjacency matrix A. t The 0th row and the 0th column.

[0127] The relative position importance matrix, B. For objects (e.g., road users) within the drivable area of ​​the scene represented in the observation frame, the importance values ​​of the elements of the relative position importance matrix B are generated using the distance between the object and the target pedestrian, calculated along the nearest edge of the road to the target pedestrian (in the direction of traffic), such as... Figure 10As shown. Specifically, the distance traveled is measured from a point on the lane edge closest to the target pedestrian, perpendicular to the road user, to a point on the edge perpendicular to the pedestrian. This distance measurement is robust to the lane's direction of travel and the road's curvature. If the object has already passed the target pedestrian, a negative value is assigned to the distance traveled (e.g., distances d2 and d4 for road users i=2 and i=4 are negative), and a positive value is assigned if the object is approaching (e.g., distances d1 and d3 for road users i=1 and i=4 are positive). Therefore, the relative positional importance of a road user is based both on the distance traveled on the road and on the road user's direction of travel relative to the target pedestrian.

[0128] Distance threshold d thresh A maximum distance from the target pedestrian can be set empirically (e.g., 20 meters), and the resulting relative importance value can be normalized accordingly. Therefore, for observation frame f... t The i-th object in the scene, located in the dable area of ​​the scene, at a distance d along the road lane, has its relative position importance value calculated as:

[0129]

[0130] For observation frame f t For any object j in the non-driveable region of the scene represented in the diagram, the corresponding matrix value can be set to: B t [0,j]=B t [j,0] = 0.5.

[0131] For any two objects m and n that are part of the same cluster, the matrix position corresponding to the relative position importance matrix Bt of the object pair can be set to Bt[m,n] = Bt[n,m] = 0. The value 0 can be assigned to all diagonal elements of matrix Bt, and the remaining unassigned values ​​are set to 1.

[0132] The distance matrix, D. The elements of the distance value matrix D are calculated using the Euclidean distances between the target pedestrian and surrounding road users. Therefore, for the k-th object in the observation frame ft,

[0133] Similar to the relative importance value matrix B, a predefined distance threshold (e.g., d) can be used. thresh =20m) normalized distance. To obtain Dt[0,k] and Dt[k,0], the following calculation can be used:

[0134]

[0135] Similar to the importance matrix, for any two objects m and n that are part of the same cluster, the interaction modeling module 914 can set Dt[m,n] = Dt[n,m] = 0, assigning 0 to all diagonal elements of Dt and assigning 1 to all remaining elements of Dt.

[0136] The adjacency matrix, A. The interaction modeling module 914 can calculate the observed frame f. t The adjacency matrix is ​​as follows:

[0137] A t =(1-B t )☉(1-D t ),

[0138] Where ⊙ represents element-wise multiplication to obtain the observation frame f t Adjacency matrix A at point t In the case of road users and target pedestrian pairs, the corresponding matrix values ​​are multi-factor relative importance weights, determined based on the distance and relative positional importance values ​​between the road users and pedestrian pairs.

[0139] In summary, when i = j or the objects belong to the same cluster, in the observation frame f t In the scenario represented in the figure, the adjacency matrix A of the i-th and j-th objects (i.e., the target pedestrian and the road user) is... t The value of [i,j] will be 1. For all edges connecting the target pedestrian and each road user, 0 ≤ A t [i,j]≤1. A is an example of A when the two objects are not part of the same cluster. t [i,j]=0.

[0140] It should be noted that the adjacency matrix A t [i,j] includes values ​​that embed information similar to that described in matrix A2 above within the context of the pedestrian action subsystem 400. Each matrix element indicates the information in the observed frame f. t This indicates whether there is a direct connection between the detected pairs of objects in the represented scene. In the case of road user type object pairs, "0" indicates no direct connection, meaning the road user pair is not included in the common cluster; "1" indicates a direct connection, meaning the road user pair is in the common cluster. Each matrix element corresponding to the pairing of road user and target pedestrian pairs will include a corresponding multi-factor relative importance weight based on the distance from the road user to the target pedestrian and the relative positional importance determined for the road user / target pedestrian pair. The relative positional importance value depends on the distance metric and the orientation of the road object relative to the target pedestrian.

[0141] When preparing to perform spatiotemporal convolution on the graph, the adjacency matrix A is formatted such that for an observation sequence (i.e., T observation frames) with a total of N unique objects (including the target pedestrian and surrounding objects) across all observation frames, the adjacency matrix A will have a dimension of N×N at each observation frame in the sequence.

[0142] If an object in the sequence does not exist in a given frame, it is not connected to any other node. This is achieved by setting all values ​​of the adjacency matrix of the corresponding row and column to zero, with the exception of setting the diagonal elements to 1. The adjacency matrix Ao of all observation frames in the target pedestrian observation sequence Ao = {A1, A2, ..., At} is... t Stacked into a 3D tensor of size N×N×T, and provided to the pedestrian motion predictor 916.

[0143] Pedestrian motion predictor 916

[0144] In an exemplary embodiment, the pedestrian action predictor 916 may include a spatiotemporal graph convolutional neural network. Spatial graph convolution is defined as Z = AVW [e.g., see T. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks”, ICLR, 2017], where A is the adjacency matrix of a frame, V is the corresponding feature matrix, and W contains the trainable weights of the spatial convolutional layer. Extending from this, as shown in

[11] , the spatiotemporal graph involves using a set of spatial graphs G. o ={G1,G2,...,GT}=(V o A o From the observation frame F in the sequence o ={f1,f2,...,f T} Construct graphs. These graphs have the same configuration at each frame, while A t The edge weights and V in t The characteristics in the data change as t ranges from 1 to T.

[0145] When executing the spatiotemporal graph convolutional component, the pedestrian motion predictor 916 can utilize the embeddings in the adjacency matrix A by using two layers of spatial convolution. o The clustering information in the convolution is used to combine the levels of indirect neighbors of each object into the convolution. Convolutions in the temporal dimension can be performed using, for example, a kernel size of 3, combining information from a given observation frame with information from its previous and subsequent observation frames.

[0146] The embedding of the target pedestrian can be represented as P, having The dimension of, where hyperparameters The value is chosen based on experience (e.g.) ), and corresponds to the output dimension.

[0147] Encode the target pedestrian and vehicle dynamics. To directly capture information related to the target pedestrian and vehicle dynamics, information can be encoded in each observation frame f. t Vector of encoded target pedestrian features and the vector of vehicle features Here, x and y are the positions of the target pedestrian and the vehicle, and vx and vy are the velocities in the global bird's-eye view reference frame. The velocities are calculated as follows: For t = 2, ..., T, where i is the target pedestrian or vehicle. At t = 1, [vx, vy] is set to [0, 0]. The speed can be multiplied by a scaling factor (e.g., 1000) to scale it to an order of magnitude similar to the global x and y coordinates.

[0148] Action classification. In the illustrative example of the pedestrian action predictor 916, to generate predictions (e.g., the probability that a pedestrian will cross the road or not), the target pedestrian's graph embedding P, pedestrian dynamics, etc., are used. and vehicle dynamics The hidden states are fed into three corresponding long short-term memory (LSTM) networks. The resulting three hidden states are cascaded and fed into an attention layer, then classified through a dense layer. For training, binary cross-entropy loss can be used in an exemplary embodiment.

[0149] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electrical hardware, or a combination of computer software and electrical hardware. Whether the function is executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be based on the corresponding processes in the above method embodiments, and will not be repeated here.

[0151] It should be understood that the disclosed systems and methods can be implemented in other ways. The units described as discrete parts may or may not be physically separate. The parts shown as units may or may not be physical units; they may be located in one location or distributed across multiple network units. Some or all units can be selected according to actual needs to achieve the purpose of the embodiments. Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0152] When these functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to instruct a computer device (which may be a personal computer, server, or network device) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes any medium capable of storing program code, such as a universal serial bus (USB) flash drive, a removable hard disk, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0153] The above descriptions are merely some specific implementations of this application and are not intended to limit the scope of protection of this application. Any variations or substitutions that can be conceived by those skilled in the art within the scope of the technology disclosed in this invention should be covered within the scope of protection of this invention. Any content from published papers cited in this invention is incorporated herein by reference.

Claims

1. A computer-based method for predicting pedestrian movements, characterized in that, The method includes: Receive time series of observation frames, each observation frame including spatial information of the target pedestrian and multiple road users; For each observation frame in the observation frame sequence: Based at least on the spatial information included in the observation frame, the target pedestrian feature set of the target pedestrian and the corresponding road user feature set of each of the plurality of road users are encoded; At least based on the spatial information included in the observation frame, a set of relative importance weights is generated, the set of relative importance weights including a corresponding relative importance weight for each of the road users, the corresponding relative importance weight indicating the relative importance of the road user to the target pedestrian, the corresponding relative importance weight of each road user being based on the distance between the road user and the target pedestrian and the relative positional importance of the road user to the target pedestrian; The future actions of the target pedestrian are predicted based on the target pedestrian feature set encoded for each of the plurality of observation frames, the corresponding road user feature set encoded for each of the plurality of observation frames, and the relative importance weight set generated for each of the plurality of observation frames. The autonomous vehicle's actions are automatically controlled based on the predicted future actions of the target pedestrian.

2. The method according to claim 1, characterized in that, The relative positional importance of each road user is based on the road user's direction of movement relative to the target pedestrian.

3. The method according to claim 2, characterized in that, The relative position of each road user is more important when they are moving toward the target pedestrian than when they are moving away from it.

4. The method according to claim 2 or 3, characterized in that, The relative positional importance of each road user is also based on the distance traveled along the road relative to the position of the target pedestrian.

5. The method according to claim 2 or 3, characterized in that, The relative positional importance of each road user is based on the distance between the road user and a reference line that extends from the position of the target pedestrian and is perpendicular to the direction of travel on the road.

6. The method according to any one of claims 1 to 3, characterized in that, For each road user, the distance between the road user and the target pedestrian is the Euclidean distance.

7. The method according to any one of claims 1 to 3, characterized in that, For each observation frame in the observation frame sequence: The target pedestrian feature set and the corresponding road user set of each of the plurality of road users are encoded based on the spatial information included in the plurality of observation frames in the sequence including the observation frames; The set of relative importance weights is generated for each road user based on the spatial information included in multiple observation frames in the sequence that includes the observation frames.

8. The method according to any one of claims 1 to 3, characterized in that, For each observation frame in the observation frame sequence: Based on the spatial information included in multiple observation frames in the sequence including the observation frames, the road user groups of the multiple road users are clustered into corresponding clusters based on behavioral similarity; The prediction of the target pedestrian's future actions is also based on the clustering.

9. The method according to claim 8, characterized in that, For each of the observation frames, a corresponding spatial map is generated, wherein, for each observation frame: The corresponding spatial graph has a target pedestrian node representing the target pedestrian, and a plurality of road user nodes each representing a corresponding one of the plurality of road users, the corresponding spatial graph being defined by: (i) a feature matrix including the encoded target pedestrian features as features of the target pedestrian node, and including the set of road user features encoded for the corresponding road user as features of the corresponding road user node; (ii) an adjacency matrix specifying: (a) a corresponding weighted connection edge between the target pedestrian node and each of the corresponding road user nodes, the target pedestrian node and each of the corresponding road user nodes corresponding to the set of relative importance weights generated for the observation frame; and (b) connection edges between each of the road user nodes included in the corresponding cluster.

10. The method according to claim 9, characterized in that, Predicting the future actions of the target pedestrian is performed using a spatiotemporal convolutional graph neural network that receives the spatial graph generated for the observed frame.

11. The method according to any one of claims 1 to 3, characterized in that, The predicted future action is either that the pedestrian will cross the road in front of the autonomous vehicle or that the pedestrian will not cross the road in front of the autonomous vehicle.

12. The method according to any one of claims 1 to 3, characterized in that, For each observation frame in the observation frame sequence: the corresponding road user feature set encoded for each of the plurality of road users includes one or more of the following: the type of the road user; the position of the road user relative to the target pedestrian; the size of the road user; the speed of the road user; and the direction of movement of the road user.

13. A processing system, characterized in that, include: One or more processor systems; One or more non-transient memories store instructions that, when executed by the one or more processor systems, cause the one or more processor systems to perform the method of any one of claims 1 to 12.

14. A non-transitory computer-readable medium, characterized in that, The storage machine-executable instructions, when executed by one or more processor systems, cause the one or more processor systems to perform the method of any one of claims 1 to 12.

Images