System and method for predicting movement of a pedestrian

Abstract

The present disclosure relates to systems and methods for predicting movements of pedestrians. A system for predicting movements of a plurality of pedestrians and related methods can include one or more processors and a memory. The memory includes an initial trajectory module, an exit point prediction module, a path rule module, and an adjustment module. The modules contain instructions that, when executed by the one or more processors, cause the one or more processors to obtain trajectories of the plurality of pedestrians, predict future exit points of the plurality of pedestrians from a scene based on the trajectories of the plurality of pedestrians, determine trajectory paths of the plurality of pedestrians based on the future exit points and at least one scene element of a map, and adjust the trajectory paths based on at least one predicted interaction between at least two of the plurality of pedestrians.

Description

Technical Field

[0001] The topics covered in this article generally involve systems and methods for predicting pedestrian movement. Background Technology

[0002] The background description provided is intended to provide a general overview of the context of this disclosure. The inventors’ work to the extent that can be described in this background section, and various aspects of the description that may not constitute prior art at the time of filing, are neither expressly nor implicitly acknowledged as prior art.

[0003] Some current vehicles are equipped with sensors that can detect objects in the environment in which the vehicle operates. Some of these detected objects include moving objects, such as other vehicles and pedestrians. In the case of pedestrians, the predicted movement of pedestrians can be utilized by several downstream components of the autonomous vehicle system, such as path planning and decision-making.

[0004] Some current methods simulate pedestrian movement dynamics by directly relying on social, scene, and / or other cues. For example, some current methods use a "social force" model that generates energy terms to avoid collisions with scene elements, other pedestrians in the scene, etc., rather than explicit modeling. Other methods leverage more data-driven approaches for dynamic modeling by relying on deep models to learn underlying mechanisms. Summary of the Invention

[0005] This section provides a general summary of the disclosure and is not intended to be a complete explanation of the full scope or all features of the disclosure.

[0006] In one embodiment, a system for predicting the movement of multiple pedestrians includes one or more processors and a memory in communication with the one or more processors. The memory includes an initial trajectory module, an exit point prediction module, a path planning module, and an adjustment module. The initial trajectory module includes instructions, when executed by the one or more processors, to cause the one or more processors to obtain the trajectories of the multiple pedestrians. The exit point prediction module includes instructions, when executed by the one or more processors, to cause the one or more processors to predict future exit points for the multiple pedestrians leaving the scene based on the trajectories of the multiple pedestrians. The path planning module includes instructions, when executed by the one or more processors, to cause the one or more processors to determine trajectory paths for the multiple pedestrians based on the future exit points and at least one scene element of a map, wherein the trajectory paths are predicted paths that the multiple pedestrians will take to reach the future exit points. The adjustment module includes instructions, when executed by the one or more processors, to cause the one or more processors to adjust the trajectory paths based on at least one predicted interaction between at least two of the multiple pedestrians.

[0007] In yet another embodiment, a method for predicting the movement of multiple pedestrians includes the following steps: obtaining the trajectories of the multiple pedestrians; predicting future exit points for the multiple pedestrians from leaving a scene based on the trajectories of the multiple pedestrians; determining trajectory paths for the multiple pedestrians based on the future exit points and at least one scene element of a map; and adjusting the trajectory paths based on at least one predicted interaction between at least two of the multiple pedestrians.

[0008] In another embodiment, a non-transitory computer-readable medium for predicting the movement of multiple pedestrians includes instructions that, when executed by one or more processors, cause the one or more processors to obtain trajectories of the multiple pedestrians, predict future exit points of the multiple pedestrians from a scene based on the trajectories of the multiple pedestrians, determine trajectory paths of the multiple pedestrians based on the future exit points and at least one scene element of a map, and adjust the trajectory paths based on at least one predicted interaction between at least two of the multiple pedestrians.

[0009] Further areas of applicability and various methods for enhancing the disclosed technology will become apparent from the provided description. The descriptions and specific examples in the summary section are for illustrative purposes only and are not intended to limit the scope of this disclosure. Attached Figure Description

[0010] The accompanying drawings, which are included in and form part of this specification, illustrate various systems, methods, and other embodiments of this disclosure. It should be understood that the illustrative element boundaries (e.g., boxes, groups of boxes, or other shapes) in the drawings represent one embodiment of a boundary. In some embodiments, one element may be designed as multiple elements, or multiple elements may be designed as one element. In some embodiments, an element representing an inner component of another element may be implemented as an outer component, and vice versa. Furthermore, the individual elements may not be drawn to scale.

[0011] Figure 1 The illustration depicts a scenario with a vehicle that includes a system for predicting pedestrian movement.

[0012] Figure 2 A diagram illustrating the predictive interaction between two pedestrians and the... Figure 1 A more detailed view of the adjusted path predicted by the system;

[0013] Figure 3 The illustration includes a more detailed view of the vehicle used by the system to predict pedestrian movement;

[0014] Figure 4 A more detailed view of the system used to predict pedestrian movement is illustrated.

[0015] Figure 5 This is a flowchart illustrating an example of the processing used by a system for predicting pedestrian movement;

[0016] Figure 6 This diagram illustrates a method for predicting pedestrian movement. Detailed Implementation

[0017] This describes a system and method for predicting the movement of one or more pedestrians. In one example, the system determines an exit point where the pedestrian leaves a scene based on their trajectory. The exit point determined by the system can be determined using a hybrid density network. Next, the system predicts the path the pedestrian will take based on the exit point and at least one scene element. For example, the scene element could be a sidewalk located between the pedestrian and the future exit point. Most likely, the pedestrian will use this sidewalk to reach the exit point. Therefore, when predicting the path the pedestrian might take, the system can consider scene elements when determining the predicted path.

[0018] The system then compares pedestrian paths to determine if any anticipated interactions exist between them, such as pedestrians simultaneously crossing each other's paths, potentially leading to a collision. If a potential interaction is determined, the system adjusts the paths to conform to social cues, road rules, etc. For example, if the system determines that the paths of two pedestrians will lead to a collision, it adjusts the paths to prevent it. The system can utilize graph neural networks to determine how to adjust the paths based on pedestrian interactions, social cues, road rules, etc.

[0019] See Figure 1 The diagram illustrates scenario 10, which includes vehicle 100. Vehicle 100 has a sensor system 120 for sensing one or more objects outside vehicle 100 and a pedestrian prediction system 170. It should be understood that scenario 10 is merely one example illustrating the pedestrian prediction system 170, which, as described below, can determine the movement of one or more pedestrians within scenario 10. In this example, vehicle 100 is traveling along road 12. Sidewalks 14 and 16 are located on either side of road 12. Sidewalks 14 and 16 may be pedestrian walkways that allow pedestrians, cyclists, and other non-vehicle-related items to travel on them.

[0020] Pedestrians 20 and 30 are also located within scene 10. Pedestrian 20 is depicted with a trajectory indicated by an arrow as trajectory 22. Trajectory 22 indicates the position, direction, and / or speed at which pedestrian 20 is moving. Similarly, pedestrian 30 also has a trajectory indicated by an arrow as trajectory 32. As before, trajectory 32 generally indicates the position, direction, and / or speed at which pedestrian 30 is moving. In this example, pedestrian 20 is attempting to cross road 12 from sidewalk 16 to sidewalk 14. As for pedestrian 30, pedestrian 30 is moving along sidewalk 14.

[0021] As described in more detail later, pedestrian prediction system 170 is capable of predicting future exit points 26 and 36 for pedestrians 20 and 30, respectively, when they leave scene 10. In this example, pedestrian prediction system 170 has predicted that pedestrian 20 will leave scene 10 at future exit point 26 based on trajectory 22 and one or more elements within scene 10. One or more elements of scene 10 may include sidewalks 14 and 16. Thus, pedestrian prediction system 170 has determined that pedestrian 20 will leave scene 10 at future exit point 26 based on trajectory 22 leading to sidewalk 16, which may be used by pedestrian 20. By using these elements, pedestrian prediction system 170 predicts future exit point 26. The path 24 taken by pedestrian 20 to future exit point 26 will also be determined by pedestrian prediction system 170. Similarly, for pedestrian 30, pedestrian prediction system 170 also predicts future exit point 36 and the path 34 that pedestrian 30 will take to reach future exit point 36.

[0022] The pedestrian prediction system 170 can also adjust the paths 24 and 34 of pedestrians 20 and 30 respectively to account for possible interactions between them as they travel along paths 24 and 34 respectively. For example, as Figure 2 As best shown, the paths 24 and 34 of pedestrians 20 and 30 lead to areas 40 that will cause a collision between them. System 170 has the ability to avoid collisions by disregarding these interactions, thereby more accurately predicting the movement of pedestrians, such as pedestrians 20 and 30. Furthermore, the movement of pedestrians 20 and / or 30, or even other objects, can be used by one or more vehicle systems to control vehicle 100. Again, the foregoing paragraphs only provide a general overview of pedestrian prediction system 170. A more detailed description of pedestrian prediction system 170 will be provided later in this disclosure.

[0023] See Figure 3 The diagram illustrates an example of vehicle 100. As used herein, "vehicle" refers to any form of powered transportation. In one or more implementations, vehicle 100 is an automobile. Although various scenarios will be described with reference to automobiles herein, it should be understood that the embodiments are not limited to automobiles. In some implementations, vehicle 100 may be, for example, any robotic device or any form of powered transportation that includes one or more automated or autonomous systems, thereby benefiting from the functions discussed herein.

[0024] In various embodiments, the automated / autonomous system or combination of systems may vary. For example, in one aspect, the automated system is a system that provides autonomous control of the vehicle at one or more automation levels, such as those defined by the Society of Automotive Engineers (SAE) (e.g., levels 0-5). Thus, the autonomous system can provide semi-autonomous or fully autonomous control as discussed with respect to autonomous driving module 160.

[0025] The vehicle 100 also includes various components. It should be understood that in various embodiments, the vehicle 100 need not have... Figure 3 All the components shown. Vehicle 100 may have Figure 3 Any combination of the various elements shown. Furthermore, vehicle 100 may have, in addition to... Figure 3 Additional components other than those shown. In some embodiments, these may be omitted. Figure 3 The vehicle 100 is implemented using one or more of the components shown. Although in Figure 3 In this diagram, each component is shown as being located within vehicle 100; however, it should be understood that one or more of these components may be located outside vehicle 100. Furthermore, the components shown may be physically separated by a considerable distance and may be provided as remote services (e.g., cloud computing services).

[0026] Some of the possible components of vehicle 100 are shown in Figure 3 In, and will be together with subsequent appendices Figure 1 This will be explained later. However, for the sake of brevity, it will be discussed later. Figure 1 and 4 -6 after providing Figure 3 The description includes numerous elements. Furthermore, it should be appreciated that, for the sake of simplicity and clarity of illustration, reference numerals are repeated in different figures where appropriate to indicate corresponding or similar elements. Additionally, the description outlines numerous specific details to provide a thorough understanding of the various embodiments described herein. Various combinations of these elements can be used to practice the various embodiments described herein.

[0027] In either case, vehicle 100 includes a pedestrian prediction system 170. The pedestrian prediction system 170 may be included within the autonomous driving module 160 of vehicle 100, or it may be a separate system as shown in the figure. See also Figure 4 The figure further illustrates one embodiment of the pedestrian prediction system 170. As shown, the pedestrian prediction system 170 includes one or more processors 110. Thus, the processor 110 may be part of the pedestrian prediction system 170, or the pedestrian prediction system 170 may access the processor 110 via a data bus or other communication path. In one or more embodiments, the processor 110 is an application-specific integrated circuit configured to implement functions associated with the initial trajectory module 250, the exit point prediction module 252, the path planning module 256, and the adjustment module 258. Typically, the processor 110 is an electronic processor capable of performing the various functions described herein, such as a microprocessor. In one embodiment, the pedestrian prediction system 170 includes a memory 210 storing the initial trajectory module 250, the exit point prediction module 252, the path planning module 256, and the adjustment module 258. The memory 210 is random access memory (RAM), read-only memory (ROM), a hard disk drive, flash memory, or other suitable memory for storing modules 250, 252, 256, and 258. Modules 250, 252, 256 and 258 are, for example, computer-readable instructions that, when executed by processor 110, cause processor 110 to perform the various functions disclosed herein.

[0028] Furthermore, in one embodiment, the pedestrian prediction system 170 includes a data warehouse 240. In one embodiment, the data warehouse 240 is an electronic data structure, such as a database, stored in memory 210 or another memory, configured with routines executable by processor 110 to analyze, provide, organize, etc., the stored data. Thus, in one embodiment, the data warehouse 240 stores data used by modules 250, 252, 256, and 258 in performing various functions. In one embodiment, the data warehouse 240 includes sensor data 242, as well as other information used by modules 250, 252, 256, and 258. Sensor data 242 may include… Figure 3 Part or all of the sensor data 119 shown and described later in this disclosure.

[0029] In addition to sensor data 242, data warehouse 240 may also include other information that modules 250, 252, 256, and 258 can utilize when performing various functions. In one example, data warehouse 240 may also include one or more artificial intelligence models. For example, data warehouse 240 may include a hybrid density network 244, a graph neural network 246, and an inverse reinforcement learning model 248. As described later, exit point prediction module 252 may utilize hybrid density network 244 to predict exit points from scene 10. Path planning module 256 may utilize inverse reinforcement learning model to determine the trajectory path of an isolated pedestrian. Adjustment module 258 may utilize graph neural network 246 to adjust the pedestrian's path to take into account any interactions with other pedestrians, such as collisions.

[0030] Therefore, the initial trajectory module 250 includes instructions, when executed by the processor 110, to enable the processor 110 to obtain the trajectories of multiple pedestrians in the scene, such as pedestrians 20 and 30 in scene 10. In one example, scene 10 may be a bird's-eye view. Scene 10 may be a static scene or may be moving based on the movement of one or more objects. In one example, a vehicle containing the pedestrian prediction system 170, such as... Figure 3 The movement of vehicle 100 can be used to determine the overall movement of scene 10 if scene 10 is a moving scene. For example, scene 10 could be a radius around vehicle 100 that moves as vehicle 100 moves.

[0031] Trajectories 22 and 32 of pedestrians 20 and 30 can be obtained from other systems and subsystems located within vehicle 100, respectively. In one example, sensor system 120 of vehicle 100 is capable of detecting the presence and movement of pedestrians 20 and / or 30. Additionally, sensor system 120 can utilize information received from one or more environmental sensors 122 to determine trajectory 22 and / or 32, which may include location, the direction in which pedestrians 20 and 30 are traveling, and the speed of pedestrians 20 and 30.

[0032] The exit point prediction module 252 may include, when executed by the processor 110, enabling the processor 110 to predict exit points, such as the future exit points 26 and 36 for pedestrians 20 and 30, respectively. Given the previous trajectories of pedestrians 20 and 30 and a clipping of the semantic map or scene 10 centered on trajectories 22 and 32, the exit point prediction module 252 may predict the future exit points 26 and 36 for pedestrians 20 and 30, respectively.

[0033] The exit point prediction module 252 can use a hybrid density network 244 to maintain a hybrid entangled normal distribution on the image or scene boundary, which approximates the future exit point given a trajectory. The hybrid density network 244 can be one or more models of a class obtained by combining a conventional neural network with a hybrid density model. The hybrid density network 244 outputs parameters of the hybrid probability distribution and weights used to combine the component distributions. In this example, the hybrid density network 244 can determine the future exit points 26 and 36 of pedestrians 20 and 30, respectively, based on their trajectories 22 and 32 and one or more scene elements.

[0034] Scene elements can include elements located within scene 10. Figure 1 In the example shown, scene elements include road 12 and sidewalks 14 and 16. It is generally understood that pedestrians such as pedestrians 20 and 30 typically use sidewalks such as sidewalks 14 and 16 and generally follow road rules. In this example, pedestrian 20 has begun crossing road 12, which does not include a crosswalk. However, based on the trajectory 22 of pedestrian 20 towards sidewalk 14, the hybrid density network 244 can determine that pedestrian 20 may continue in the same direction to sidewalk 14 and continue along sidewalk 14 to a future exit point 26. Similarly, the hybrid density network 244 can determine, based on the trajectory 32 of pedestrian 30, that pedestrian 30 may continue along sidewalk 14 and leave scene 10 at a future exit point 36.

[0035] The path planning module 256 includes instructions, when executed by the processor 110, to cause the processor 110 to predict the path of an isolated pedestrian. Furthermore, future exit points, such as future exit points 26 and 36, are sampled from the exit point prediction module 252 and fed into the path planning module 256. The path planning module 256 plans a human-like trajectory (or path) for the pedestrian to achieve the pedestrian's goal of reaching the exit point predicted by the exit point prediction module 252.

[0036] In this example, the path of an isolated pedestrian can be interpreted as a prediction of the path the pedestrian will take. For example, see [link to previous section]. Figure 1 Pedestrian 20 has been determined by path planning module 256 to travel along path 24, while pedestrian 30 has been determined to travel along path 34. Path planning module 256 may utilize trajectories 22 and 32 previously determined by initial trajectory module 250, and future exit points 26 and / or 36 predicted by exit point prediction module 252. In addition to these inputs, path planning module 256 may also utilize other inputs, such as road rules. For example, it is generally assumed that pedestrians will follow road rules, such as using appropriate locations to cross road 12, using pedestrian crossings, and following road signals such as stop signs and traffic lights.

[0037] The path planning module 256 can utilize the inverse reinforcement learning model 248. Inverse reinforcement learning is a machine learning architecture that solves the inverse problem of reinforcement learning. Furthermore, inverse reinforcement learning is about learning the purpose, value, or reward of an agent by observing its behavior. For example, traditional reinforcement learning settings typically require the goal to learn a decision-making process to produce behavior that maximizes a predefined reward function. Inverse reinforcement learning typically reverses this problem and instead attempts to extract the reward function from the observed behavior of agents such as pedestrians 20 and 30.

[0038] The adjustment module 258 includes instructions, when executed by the processor 110, to adjust the trajectory paths determined by the path planning module 256 to take into account the interactions between one or more behaviors. Given a long-term trajectory for each pedestrian present in the scene, all these trajectories are negotiated and adjusted to avoid collisions and / or follow general social cues and road rules. This is achieved by embedding trajectories as node features in a graph neural network 246, which follows a message-passing algorithm to adjust these predictions.

[0039] For example, see Return Figure 1As previously explained, if pedestrians 20 and 30's paths 24 and 34 remain unchanged, they will collide. To determine whether a collision will occur, adjustment module 258 configures processor 110 to determine whether pedestrian paths 24 and 34 intersect. Furthermore, to determine whether paths 24 and 34 intersect, adjustment module 258 also configures processor 110 to determine whether pedestrians will collide if they continue along paths 24 and 34 at a predicted rate. In some cases, pedestrian paths may intersect, but due to the pedestrians' positions and speeds, a collision will not occur. However, in other cases, the pedestrians' speeds and positions, as well as overlapping path indications, will lead to a collision.

[0040] In the event of a predicted interaction between pedestrians, such as a collision, the adjustment module 258 adjusts the predicted trajectory paths of pedestrians 20 and 30, for example, the predicted trajectory paths 24 and 34. The adjustment module 258 may utilize a graph neural network 246 to adjust the pedestrian trajectory paths to avoid the expected collision, since pedestrians typically do not intentionally collide with each other.

[0041] Graph Neural Networks (GNNs) are a type of neural network that operates directly on graph structures. Therefore, GNNs can operate on graphs with more complex geometries and topologies. This can include social networks, 3D meshes, and physical systems. Thus, GNNs can be used to negotiate social interactions between pedestrians to avoid collisions and to more accurately predict the movement of pedestrians, such as pedestrians 20 and 30. See also... Figure 2 The adjustment module 258 has adjusted paths 24 and 34 respectively to avoid direct collisions between pedestrians 20 and 30. By utilizing the graph neural network 246's ability to consider social interactions (such as pedestrians' expectations of avoiding collisions with each other and other social cues), the pedestrian prediction system 170 can more accurately predict pedestrian movement within the scene.

[0042] For a better illustration of the different types of artificial intelligence networks and models that might be used, see [link to relevant documentation]. Figure 5 In this example, the scene is static; however, the scene does not have to be static and can move as the vehicle moves using motion compensation. Furthermore, Figure 5 The diagram illustrates the exit point prediction module 252, the path planning module 256, and the adjustment module 258. Additionally, Figure 5The diagram illustrates the information flow where the exit point prediction module 252, path planning module 256, and adjustment module 258 can be daisy-chained together. The output of the exit point prediction module 252 is fed into the path planning module 256, which in turn feeds into the adjustment module 258. Thus, in this example, three different types of artificial intelligence models are utilized. Furthermore, the exit point prediction module 252 utilizes a hybrid density network 224, the path planning module 256 utilizes an inverse reinforcement learning model 248, and the adjustment module 258 utilizes a graph neural network 246.

[0043] The final output from adjustment module 258 then becomes the short-term predicted trajectory of the pedestrian. Note that in this way, pedestrian prediction system 170 considers not only short-term information such as social cues, but also longer-term signals such as targets and static scene elements.

[0044] See Figure 6 The figure illustrates method 300 for predicting pedestrian movement. (The following will be discussed...) Figure 3 Vehicle 100 and Figure 4 Method 300 is explained from the perspective of pedestrian prediction system 170. However, this is only one example of implementing method 300. Although method 300 is discussed in conjunction with pedestrian prediction system 170, it should be understood that method 300 is not limited to implementation within pedestrian prediction system 170; rather, this is merely an example of a system in which method 300 can be implemented.

[0045] Method 300 begins at step 302, where the initial trajectory module 250 enables the processor 110 to acquire trajectories of multiple pedestrians. In this example, the initial trajectory module 250 may receive one or more trajectories, such as trajectories 22 and 32 of pedestrians 20 and 30. Trajectories 22 and 32 of pedestrians 20 and 30 may be obtained from other systems and subsystems located within the vehicle 100. In one example, the sensor system 120 of the vehicle 100 is capable of detecting the presence and movement of pedestrians 20 and / or 30. Additionally, the sensor system 120 may utilize information received from one or more environmental sensors 122 to determine trajectories 22 and / or 32, which may include location, the direction in which pedestrians 20 and 30 are traveling, and the speed of pedestrians 20 and 30.

[0046] In step 304, the exit point prediction module 252 causes the processor 110 to determine one or more exit points, such as future exit points 26 and 36 associated with pedestrians 20 and 30. As previously described, the exit point prediction module 252 can use a hybrid density network 244 to maintain a hybrid entangled normal distribution on the image or scene boundary, which approximates the future exit points given a trajectory. The hybrid density network 244 can be one or more models of a class obtained by combining a conventional neural network with a hybrid density model. The hybrid density network 244 outputs parameters of the hybrid probability distribution and weights for combining the component distributions. In this example, the hybrid density network 244 can determine the future exit points 26 and 36 of pedestrians 20 and 30 based on their trajectories 22 and 32 and one or more scene elements.

[0047] Scene elements can include elements located within scene 10. Figure 1 In the example shown, scene elements include road 12 and sidewalks 14 and 16. It is generally understood that pedestrians such as pedestrians 20 and 30 typically use sidewalks such as sidewalks 14 and 16 and generally follow road rules. In this example, pedestrian 20 has begun crossing road 12, which does not include a crosswalk. However, based on the trajectory 22 of pedestrian 20 towards sidewalk 14, the hybrid density network 244 can determine that pedestrian 20 may continue in the same direction to sidewalk 14 and continue along sidewalk 14 to a future exit point 26. Similarly, the hybrid density network 244 can determine, based on the trajectory 32 of pedestrian 30, that pedestrian 30 may continue along sidewalk 14 and leave scene 10 at a future exit point 36.

[0048] In step 306, the path rule module 256 causes the processor 110 to determine the trajectory paths of the plurality of pedestrians 20 and 30 based on future exit points 26 and 36, and at least one element of scenario 10. Furthermore, future exit points such as future exit points 26 and 36 are sampled from the exit point prediction module 252 and fed into the path planning module 256. The path planning module 256 plans human-like trajectories (or paths) for the pedestrians to achieve the pedestrians' goal of reaching the exit points predicted by the exit point prediction module 252.

[0049] In this example, the path of an isolated pedestrian can be interpreted as a prediction of the path the pedestrian will take. For example, see [link to previous section]. Figure 1Pedestrian 20 has been determined by path planning module 256 to travel along path 24, while pedestrian 30 has been determined to travel along path 34. Path planning module 256 may utilize trajectories 22 and 32 previously determined by initial trajectory module 250, and future exit points 26 and / or 36 predicted by exit point prediction module 252. In addition to these inputs, path planning module 256 may also utilize other inputs, such as road rules. For example, it is generally assumed that pedestrians will follow road rules, such as using appropriate locations to cross road 12, using pedestrian crossings, and following road signals such as stop signs and traffic lights.

[0050] In step 308, adjustment module 258 enables processor 110 to determine whether any predicted interaction exists between pedestrians 20 and 30. For example, by using the positions of pedestrians 20 and 30, trajectories 22 and 32, and paths 24 and 34, respectively, adjustment module 258 enables processor 110 to determine the possibility of a collision or interaction between pedestrians 20 and 30. If no interaction is detected between any pedestrians, the previously calculated trajectory path can be output, as shown in step 310. In one example, the previously calculated trajectory path can be output to one or more vehicle systems or subsystems, such as autonomous driving module 160.

[0051] If it is determined that pedestrians may interact with each other, the method proceeds to step 312, where adjustment module 258 causes processor 110 to adjust paths 24 and 34 based on the predicted interaction between pedestrians 20 and 30. Adjustment module 258 adjusts the pedestrian trajectory paths, such as the predicted paths 24 and 34 for pedestrians 20 and 30 respectively. Adjustment module 258 may utilize graph neural network 246 to adjust the pedestrian trajectory paths to avoid, as expected, collisions, since pedestrians typically do not intentionally collide with each other. Once paths 24 and 34 have been adjusted by adjustment module 258, the trajectory paths can be output to one or more vehicle systems or subsystems, such as autonomous driving module 160, as shown in step 310.

[0052] Now let's discuss this in great detail. Figure 3This serves as an illustrative environment in which the systems and methods disclosed herein may operate. In one or more embodiments, vehicle 100 is an autonomous vehicle. As used herein, "autonomous vehicle" refers to a vehicle operating in an autonomous mode. "Autonomous mode" refers to controlling vehicle 100, navigating and / or maneuvering vehicle 100 along a driving route using one or more computing systems with minimal or no input from a human driver. In one or more embodiments, vehicle 100 is highly automated or fully automated. In one embodiment, vehicle 100 is configured with one or more semi-autonomous operating modes in which one or more computing systems perform a portion of the navigation and / or maneuvering of vehicle 100 along a driving route, and a vehicle operator (i.e., driver) provides input to the vehicle to perform a portion of the navigation and / or maneuvering of vehicle 100 along a driving route. Such semi-autonomous operation may include supervisory control, such as that implemented by pedestrian prediction system 170, to ensure that vehicle 100 remains within prescribed state constraints.

[0053] Vehicle 100 may include one or more processors 110. In one or more embodiments, processor 110 may be the main processor of vehicle 100. For example, processor 110 may be an electronic control unit (ECU). Vehicle 100 may include one or more data warehouses 115 for storing one or more types of data. Data warehouse 115 may include volatile and / or non-volatile memory. Examples of data warehouse 115 include RAM (random access memory), flash memory, ROM (read-only memory), PROM (programmable read-only memory), EPROM (erasable programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), registers, disks, optical disks, hard disks, or any other suitable storage medium, or any combination thereof. Data warehouse 115 may be a component of processor 110, or data warehouse 115 may be operatively connected to processor 110 for its use. The term "operatively connected" as used in this specification may include direct or indirect connections, including connections without direct physical contact.

[0054] In one or more embodiments, the one or more data warehouses 115 may include map data 116. Map data 116 may include maps of one or more geographic areas. In some cases, map data 116 may include information or data about roads, traffic control facilities, road signs, structures, features, and / or landmarks within the one or more geographic areas. Map data 116 may be in any suitable form. In some cases, map data 116 may include a bird's-eye view of the area. In some cases, map data 116 may include a ground view of the area, including a 360° ground view. Map data 116 may include measurements, dimensions, distances, and / or information about one or more items included in map data 116 and / or relative to other items included in map data 116. Map data 116 may include digital maps with information about road geometry. Map data 116 may be of high quality and / or very detailed.

[0055] In one or more embodiments, map data 116 may include one or more topographic maps 117. Topographic map 117 may include information about the ground, topography, roads, surfaces, and / or other features of one or more geographic areas. Topographic map 117 may include elevation data for the one or more geographic areas. Map data 116 may be of high quality and / or very detailed. Topographic map 117 may define one or more land surfaces, which may include paved roads, unpaved roads, land, and other things that define the land surface.

[0056] In one or more schemes, map data 116 may include one or more static obstacle maps 118. Static obstacle maps 118 may include information about one or more static obstacles located within one or more geographic areas. A “static obstacle” is a physical object whose location remains unchanged or substantially unchanged over a period of time, and / or whose size remains unchanged or substantially unchanged over a period of time. Examples of static obstacles include trees, buildings, curbs, fences, railings, median strips, utility poles, statues, monuments, signs, benches, furniture, mailboxes, large boulders, and hills. Static obstacles may be objects extending above the ground plane. One or more static obstacles included in static obstacle maps 118 may have location data, size data, latitude data, material data, and / or other data associated with them. Static obstacle maps 118 may include measurements, dimensions, distances, and / or information about one or more static obstacles. Static obstacle maps 118 may be of high quality and / or very detailed. Static obstacle maps 118 may be updated to reflect changes within the mapping area.

[0057] One or more data warehouses 115 may include sensor data 119. In this context, "sensor data" means any information about the sensors equipped on vehicle 100, including information about the capabilities of such sensors and other information. As described below, vehicle 100 may include sensor system 120. Sensor data 119 may be associated with one or more sensors of sensor system 120. For example, in one or more embodiments, sensor data 119 may include information about one or more LiDAR sensors 124 of sensor system 120.

[0058] In some cases, at least a portion of the map data 116 and / or sensor data 119 may be located in one or more data warehouses 115 located on the vehicle 100. Alternatively, or additionally, at least a portion of the map data 116 and / or sensor data 119 may be located in one or more data warehouses 115 located remotely from the vehicle 100.

[0059] As described above, vehicle 100 may include sensor system 120. Sensor system 120 may include one or more sensors. “Sensor” means any device, component, and / or system capable of detecting and / or sensing something. The one or more sensors may be configured to detect and / or sense in real time. The term “real time” as used herein means that the user or system feels it is sufficiently immediate for a particular processing or decision to be performed, or that the processor is able to keep up with the processing response level of some external processing.

[0060] In a configuration where sensor system 120 includes multiple sensors, the sensors can operate independently of each other. Alternatively, two or more sensors can operate in combination. In this case, the two or more sensors can form a sensor network. Sensor system 120 and / or one or more sensors can be operatively connected to processor 110, data warehouse 115, and / or other components of vehicle 100 (including...). Figure 3 (Any element shown). The sensor system 120 can acquire data on at least a portion of the external environment of the vehicle 100 (e.g., nearby vehicles).

[0061] Sensor system 120 may include any suitable type of sensor. Various examples of different types of sensors will be described herein. However, it should be understood that the embodiments are not limited to the specific sensors described. Sensor system 120 may include one or more vehicle sensors 121. Vehicle sensors 121 may detect, determine, and / or sense information about vehicle 100 itself. In one or more embodiments, vehicle sensors 121 may be configured to detect and / or sense changes in the position and orientation of vehicle 100, for example, based on inertial acceleration. In one or more embodiments, vehicle sensors 121 may include one or more accelerometers, one or more gyroscopes, inertial measurement units (IMUs), dead reckoning systems, global navigation satellite systems (GNSS), global positioning systems (GPS), navigation systems 147, and / or other suitable sensors. Vehicle sensors 121 may be configured to detect and / or sense one or more characteristics of vehicle 100. In one or more embodiments, vehicle sensors 121 may include a speedometer that determines the current speed of vehicle 100.

[0062] Alternatively, sensor system 120 may include one or more environmental sensors 122 configured to acquire and / or sense driving environment data. "Driving environment data" includes data or information about the external environment in which the autonomous vehicle is located, or one or more portions thereof. For example, the one or more environmental sensors 122 may be configured to detect, quantify, and / or sense obstacles in at least a portion of the external environment of vehicle 100, and / or information / data about such obstacles. Such obstacles may be stationary objects and / or moving objects. The one or more environmental sensors 122 may be configured to detect, measure, quantify, and / or sense other things in the external environment of vehicle 100, such as lane markings, signs, traffic lights, traffic signs, lane lines, pedestrian crossings, curbs adjacent to vehicle 100, objects outside the road, etc.

[0063] Various examples of sensors for sensor system 120 will be described herein. The illustrated sensors may be part of one or more environmental sensors 122 and / or one or more vehicle sensors 121. However, it should be understood that the embodiments are not limited to the specific sensors described.

[0064] For example, in one or more embodiments, sensor system 120 may include one or more radar sensors 123, one or more lidar sensors 124, one or more sonar sensors 125, and / or one or more cameras 126. In one or more embodiments, the one or more cameras 126 may be high dynamic range (HDR) cameras or infrared (IR) cameras.

[0065] Vehicle 100 may include an input system 130. An "input system" includes any device, component, system, element, or apparatus, or a group of said devices, components, systems, elements, or apparatuses, that enables information / data to be input into the machine. Input system 130 may receive input from vehicle passengers (e.g., a driver or passenger). Vehicle 100 may include an output system 135. An "output system" includes any device, component, or apparatus, or a group of said devices, components, or apparatuses, that enables information / data to be presented to vehicle passengers (e.g., a person, vehicle passenger, etc.).

[0066] Vehicle 100 may include one or more vehicle systems 140. Figure 3 Various examples of the one or more vehicle systems 140 are illustrated herein. However, vehicle 100 may include more, fewer, or different vehicle systems. It should be appreciated that although specific vehicle systems are defined separately, each or any system or its components may be otherwise combined or separated within vehicle 100 via hardware and / or software. Vehicle 100 may include a propulsion system 141, a braking system 142, a steering system 143, a throttle system 144, a transmission system 145, a signaling system 146, and / or a navigation system 147. Each of these systems may include one or more devices, components, and / or combinations thereof that are currently known or developed hereafter.

[0067] Navigation system 147 may include one or more currently known or later developed devices, applications, and / or combinations thereof configured to determine the geographic location of vehicle 100 and / or determine the driving route of vehicle 100. Navigation system 147 may include one or more mapping applications for determining the driving route of vehicle 100. Navigation system 147 may include a Global Positioning System, a Local Positioning System, or a Geolocation System.

[0068] Processor 110, pedestrian prediction system 170, and / or autonomous driving module 160 may be operatively connected to communicate with various vehicle systems 140 and / or their respective components. For example, return Figure 3 The processor 110 and / or the autonomous driving module 160 can communicate to send and / or receive information from the various vehicle systems 140, thereby controlling the movement, speed, handling, heading, and direction of the vehicle 100. The processor 110, the pedestrian prediction system 170, and / or the autonomous driving module 160 can control some or all of these vehicle systems 140, thus enabling partial or full autonomy.

[0069] Processor 110, pedestrian prediction system 170, and / or autonomous driving module 160 may be operatively connected to communicate with various vehicle systems 140 and / or their respective components. For example, return Figure 3The processor 110, pedestrian prediction system 170, and / or autonomous driving module 160 can communicate to send and / or receive information from various vehicle systems 140, thereby controlling the movement, speed, handling, heading, and direction of vehicle 100. The processor 110, pedestrian prediction system 170, and / or autonomous driving module 160 can control some or all of these vehicle systems 140.

[0070] Processor 110, pedestrian prediction system 170, and / or autonomous driving module 160 are operable to control the navigation and / or maneuvering of vehicle 100 by controlling one or more of the vehicle systems 140 and / or components thereof. For example, when operating in autonomous mode, processor 110, pedestrian prediction system 170, and / or autonomous driving module 160 may control the direction and / or speed of vehicle 100. Processor 110, pedestrian prediction system 170, and / or autonomous driving module 160 may cause vehicle 100 to accelerate (e.g., by increasing the supply of fuel to the engine), decelerate (e.g., by decreasing the supply of fuel to the engine and / or by applying braking), and / or change direction (e.g., by turning both front wheels). As used herein, "cause" means directly or indirectly causing, forcing, directing, commanding, instructing, and / or permitting an event or action to occur, or at least being in a state in which such event or action may occur.

[0071] Vehicle 100 may include one or more actuators 150. Actuators 150 may be any element or combination of elements operable to modify, adjust, and / or alter one or more of the vehicle systems 140 or components thereof in response to receiving signals or other inputs from processor 110 and / or autonomous driving module 160. Any suitable actuator may be used. For example, the one or more actuators 150 may include electric motors, pneumatic actuators, hydraulic pistons, relays, solenoids, and / or piezoelectric actuators, to name just a few possibilities.

[0072] Vehicle 100 may include one or more modules, at least some of which are described herein. A module may be implemented, when executed by processor 110, as computer-readable program code implementing one or more of the various processes described herein. One or more modules may be components of processor 110, or one or more modules may be executed on other processing systems operatively connected to processor 110, and / or distributed among said other processing systems. A module may include instructions (e.g., program logic) executable by one or more processors 110. Alternatively, one or more data warehouses 115 may contain such instructions.

[0073] In one or more embodiments, one or more modules described herein may include artificial or computational intelligence elements, such as neural networks, fuzzy logic, or other machine learning algorithms. Furthermore, in one or more embodiments, one or more modules may be distributed among multiple modules described herein. In one or more embodiments, two or more modules described herein may be combined into a single module.

[0074] Vehicle 100 may include one or more autonomous driving modules 160. Autonomous driving module 160 may be configured to receive data from sensor system 120, and / or any other type of system capable of capturing information related to vehicle 100 and / or its external environment. In one or more scenarios, autonomous driving module 160 may use such data to generate one or more driving scenario models. Autonomous driving module 160 may determine the position and speed of vehicle 100. Autonomous driving module 160 may determine the position of obstacles, obstacles, or other environmental features, including traffic signs, trees, bushes, adjacent vehicles, pedestrians, etc.

[0075] The autonomous driving module 160 may be configured to receive and / or determine the location information of obstacles in the external environment of the vehicle 100, so that the processor 110 and / or one or more of the modules described herein may be used to estimate the position and orientation of the vehicle 100, the vehicle position in a global coordinate system based on signals from multiple satellites, or any other data and / or signals that may be used to determine the current state of the vehicle 100 or to determine the position of the vehicle 100 relative to its environment for the purpose of creating a map or determining the position of the vehicle 100 with respect to map data.

[0076] The autonomous driving module 160 can be configured independently or in combination with the pedestrian prediction system 170 to determine the driving path, the current autonomous driving maneuver of the vehicle 100, the future autonomous driving maneuver, and / or modifications to the current autonomous driving maneuver based on data acquired by the sensor system 120, the driving scenario model, and / or data from any other suitable source, such as the determination based on sensor data 250 implemented by the transmitting module 230. “Driving maneuver” means one or more actions that affect the movement of the vehicle. Examples of driving maneuvers include: acceleration, deceleration, braking, steering, moving laterally along the vehicle 100, changing lanes, merging lanes, and / or reversing, to name just a few. The autonomous driving module 160 can be configured to perform the determined driving maneuver. The autonomous driving module 160 can directly or indirectly cause such autonomous driving maneuver to be performed. As used herein, “cause” means directly or indirectly causing, commanding, instructing, and / or permitting an event or action to occur, or at least placing it in a state where such an event or action could occur. The autonomous driving module 160 can be configured to perform various vehicle functions and / or send data to the vehicle 100 or one or more of its systems (e.g., one or more of the vehicle systems 140), receive data from the vehicle 100 or one or more of its systems, interact with the vehicle 100 or one or more of its systems, and / or control the vehicle 100 or one or more of its systems.

[0077] Detailed embodiments are disclosed herein. However, it should be understood that the disclosed embodiments are merely examples. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but rather serve as the basis for the claims and as a representative basis for teaching those skilled in the art to adopt the various aspects herein differently in substantially any reasonably detailed structure. Furthermore, the terms and phrases used herein are not limiting, but rather provide an understandable description of various possible implementations. Figure 1-6 Various embodiments are illustrated herein; however, the embodiments are not limited to the illustrated structures or applications.

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments. In this regard, each block in the flowchart or block diagram may represent a module, segment, or code portion containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions described in the blocks may not occur in the order shown in the drawings. For example, two blocks shown successively may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved.

[0079] The systems, components, and / or processes described above can be implemented in hardware, or a combination of hardware and software, and can be implemented centrally in a single processing system or distributed, in the case of a distributed implementation, with different components distributed among several interconnected processing systems. Any type of processing system or other device suitable for performing the various methods described herein is suitable. A typical combination of hardware and software can be a processing system having computer-usable program code that, when loaded and executed, controls the processing system to perform the methods described herein. Systems, components, and / or processes can also be embedded in a computer-readable storage medium, such as a computer program product or other data program storage device, which is machine-readable and tangibly contains a program that can be executed by a machine to perform the methods and processes described herein. These elements can also be embedded in an application product that contains all the features that make the implementation of the methods described herein possible and, when loaded into a processing system, is capable of executing these methods.

[0080] Furthermore, the various embodiments described herein may take the form of a computer program product contained in one or more computer-readable media having computer-readable program code contained thereon, for example, stored thereon. Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The phrase "computer-readable storage medium" means a non-transitory storage medium. A computer-readable storage medium may be, for example, (but not limited to) an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination thereof. More specific examples (not an exhaustive list) of computer-readable storage media include: portable computer disks, hard disk drives (HDDs), solid-state drives (SSDs), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable optical disc read-only memory (CD-ROM), digital versatile optical disc (DVD), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium may be any tangible medium that can contain or store a program for use by, or in conjunction with, an instruction execution system, device, or apparatus.

[0081] Typically, modules as used herein include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific data type. Additionally, memory typically stores the aforementioned modules. The memory associated with a module may be a buffer or cache embedded within a processor, RAM, ROM, flash memory, or other suitable electronic storage media. In further additional aspects, the modules contemplated in this disclosure are implemented as application-specific integrated circuits (ASICs), system-on-a-chip (SoC) hardware components, programmable logic arrays (PLAs), or other suitable hardware components embedded with a defined configuration set (e.g., instructions) for performing the disclosed functions.

[0082] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including (but not limited to) wireless, wired, fiber optic, cable, RF, etc., or any suitable combination thereof. The computer program code for performing operations of various aspects of the invention may be written in one or more programming languages, including any combination of object-oriented programming languages ​​such as Java™, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may run entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The singular form used herein is defined as one or more. The term "multiple" used herein is defined as two or more. The term "another" used herein is defined as at least a second or more. The terms "including" and / or "having" used herein are defined as including (i.e., open-ended language). The phrase "at least one of... and..." used herein refers to and includes any possible combination of one or more of the associated enumerated items. For example, the phrase "at least one of A, B, and C" includes only A, only B, only C, or any combination thereof (e.g., AB, AC, BC, or ABC).

[0084] The various aspects described herein may be embodied in other forms without departing from their spirit or essential nature. Therefore, in indicating the scope of this disclosure, reference should be made to the following claims rather than the foregoing description.

Claims

1. A system for predicting the movement of multiple pedestrians, comprising: One or more processors; A memory communicating with the one or more processors, the memory having an initial trajectory module, an exit point prediction module, a path planning module, and an adjustment module; The initial trajectory module includes instructions that, when executed by the one or more processors, cause the one or more processors to obtain the trajectories of the plurality of pedestrians; The exit point prediction module includes instructions, when executed by the one or more processors, to predict the future exit points of the multiple pedestrians leaving the scene based on the trajectories of the multiple pedestrians and a semantic map of the scene, wherein the future exit points are predicted at the boundaries of the scene. The path planning module includes instructions, when executed by the one or more processors, to cause the one or more processors to determine the trajectory path of the plurality of pedestrians based on the future exit point and at least one scene element of the semantic map, wherein the trajectory path is a human-like path predicted for the plurality of pedestrians to reach the future exit point. and The adjustment module includes, when executed by the one or more processors, instructions to adjust the trajectory paths by inputting the trajectory paths as node features into a graph neural network in response to predicting collision interactions between at least two of the plurality of pedestrians. The graph neural network performs message passing to jointly adjust the trajectory paths of the plurality of pedestrians, thereby avoiding collisions while adhering to social cues and road rules.

2. The system according to claim 1, wherein the initial trajectory module further includes instructions, when executed by the one or more processors, to cause the one or more processors to obtain the trajectories of the plurality of pedestrians by using a hybrid density network model.

3. The system according to claim 1, wherein the adjustment module further includes, when executed by the one or more processors, instructions for the one or more processors to adjust the trajectory path by utilizing a graph neural network model based on at least one predicted interaction between at least two of the plurality of pedestrians.

4. The system according to claim 3, wherein the trajectories of the plurality of pedestrians are embedded in the graph neural network model as node features of the graph neural network model.

5. The system according to claim 1, wherein the scene is a fixed scene.

6. The system according to claim 1, wherein the scene moves based on the movement of the vehicle itself.

7. A method for predicting the movement of multiple pedestrians, the method comprising the following steps: Obtain the trajectories of the multiple pedestrians; Based on the trajectories of the multiple pedestrians and the semantic map of the scene, the future exit points of the multiple pedestrians leaving the scene are predicted, wherein the future exit points are predicted at the boundaries of the scene. Based on the future exit point and at least one scene element of the semantic map, determine the trajectory path of the plurality of pedestrians, wherein the trajectory path is a human-like path predicted for the plurality of pedestrians to reach the future exit point. and In response to predicting collision interactions between at least two of the plurality of pedestrians, the trajectory paths are adjusted by inputting them as node features into a graph neural network. The graph neural network performs message passing to jointly adjust the trajectory paths of the plurality of pedestrians, thereby avoiding collisions while adhering to social cues and road rules.

8. The method according to claim 7, wherein the step of obtaining the trajectories of the plurality of pedestrians is performed by a hybrid density network model.

9. The method according to claim 7, wherein the scene is a fixed scene.

10. The method according to claim 7, wherein the scene moves based on the movement of the self-vehicle.

11. A non-transitory computer-readable medium for predicting the movement of multiple pedestrians, the non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: Obtain the trajectories of the multiple pedestrians; Based on the trajectories of the multiple pedestrians and the semantic map of the scene, the future exit points of the multiple pedestrians leaving the scene are predicted, wherein the future exit points are predicted at the boundaries of the scene. In response to predicting collision interactions between at least two of the plurality of pedestrians, a trajectory path of the plurality of pedestrians is determined based on the future exit point and at least one scene element of the semantic map, wherein the trajectory path is a human-like path predicted for the plurality of pedestrians to reach the future exit point. and The trajectory path is adjusted by inputting it as node features into a graph neural network. The graph neural network performs message passing to jointly adjust the trajectory paths of the multiple pedestrians, thereby avoiding collisions while adhering to social cues and road rules.

12. The non-transitory computer-readable medium according to claim 11, wherein the scene moves based on the movement of the self-vehicle.

Images