Pedestrian trajectory prediction and intention estimation method based on double-flow lstm

Abstract

The application relates to the technical field of trajectory prediction, and discloses a pedestrian trajectory prediction and intention estimation method based on a double-flow LSTM, which comprises the following steps: S1: overall architecture design, an Encoder tra -Decoder tra architecture is used to predict the future trajectory of the pedestrian by using the direction and historical trajectory of the pedestrian; S2: foot point; S3: relative displacement, the position of the pedestrian in the picture is expressed by pixel coordinates, whether it is a boundary frame corner point or a foot point, which only represents the absolute position of the pedestrian in the picture and is an independent existence, and lacks association with context information. The position relationship between the pedestrian and the car in the application has high correlation, so the relative displacement is used to represent the relative position of the pedestrian, which can simulate the dynamic change of the distance and position between the pedestrian and the car, enhances the interaction between the pedestrian and the car, and further improves the accuracy of the pedestrian intention estimation by using the future trajectory of the pedestrian.

Description

Technical Field

[0001] This invention relates to the field of trajectory prediction technology, specifically to a pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM. Background Technology

[0002] To address the safety concerns posed by pedestrians, researchers have proposed various methods, such as pedestrian detection. However, pedestrian movement is highly random, especially when a pedestrian suddenly appears in front of a vehicle. Therefore, even if a pedestrian is detected, the car may not have enough time to avoid them. Pedestrian trajectory prediction, on the other hand, can predict a pedestrian's future trajectory by observing historical trajectories, thus allowing sufficient time for the vehicle to avoid them. Furthermore, trajectories can reveal a pedestrian's intentions, which directly reflect their actual behavior. Therefore, by predicting pedestrian trajectories and estimating their intentions, vehicles can make informed decisions, allowing more time for maneuver and significantly reducing the occurrence of traffic accidents.

[0003] Although pedestrian trajectory and intent are of great practical significance, most existing methods can only solve one of them, and cannot predict trajectory and estimate intent simultaneously in one method. Solving the two problems separately leads to the loss of correlation between trajectory and intent, which is not conducive to understanding pedestrian behavior. In addition, since the two are closely related, the accuracy of trajectory prediction and intent estimation will decrease when they are solved separately. Therefore, a pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM is proposed. Summary of the Invention

[0004] (a) Technical problems to be solved

[0005] To address the shortcomings of existing technologies, this invention provides a pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM. This method solves the problem that most existing methods can only solve one of the two aspects, and cannot simultaneously predict the trajectory and estimate the intent in one method. Solving the two problems separately leads to a lack of correlation between the trajectory and the intent, which is not conducive to understanding pedestrian behavior. In addition, since the two are closely related, the accuracy of trajectory prediction and intent estimation will decrease when they are solved separately.

[0006] (II) Technical Solution

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] The pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM includes the following steps:

[0009] S1: Overall architecture design, building an Encoder tra -Decoder traThe LSTM architecture uses the pedestrian's direction and historical trajectory to predict the pedestrian's future trajectory;

[0010] S2: Footpoint. In most pedestrian trajectory prediction and intent estimation tasks, the pedestrian's position is typically represented by the top-left corner (x, y) of the bounding box. tl ,y tl ) and bottom right corner (x br ,y br The pixel coordinates of ) are used to represent the bounding_box, which is described as bounding_box = [(x tl ,y tl ),(x br ,y br Some studies treat foot coordinates as pedestrian positions, defining them as the midpoints of the bounding box's baseline, which can be formally represented as...

[0011] S3: Relative Displacement. The position of a pedestrian in the image is represented by pixel coordinates, whether it is the corner point of the bounding box or the foot point. It only represents the absolute position of the pedestrian in the image. It is an independent existence and lacks association with contextual information. However, the relative positional relationship between the pedestrian and the car often determines a certain behavior of the pedestrian. Relative displacement is used to represent this relative value to simulate the relative positional relationship between pedestrians and vehicles.

[0012] S4: Pedestrian trajectory prediction stream. The pedestrian trajectory prediction stream mainly uses the observed historical trajectory and the pedestrian direction at each moment as input. These features are encoded in the encoder, and the decoder of the trajectory prediction stream decodes the encoded information. The output of the decoder at each moment is used as the future trajectory record of the pedestrian.

[0013] S5: Pedestrian Intent Estimation Stream. In this stream, the encoder takes the historical relative displacements of pedestrians and vehicles and the encoder output from the prediction stream as input, and encodes these features. In the intent estimation stream, the decoder receives the encoded features and future relative displacements. The final state of the decoder is the pedestrian intent estimation result.

[0014] As a further aspect of the present invention, S3 includes a relative displacement generation unit. Since the pedestrian image is taken from a first-person perspective, the distance between the pedestrian and the vehicle is mainly represented in the image as a lateral distance. First, two boundary lines are defined in the image: the left boundary line l1 and the right boundary line l2. The width between the two boundary lines can represent the width of the vehicle, or the area the car will pass through in the image. Because the upper three-quarters of the image is far from the vehicle and the pedestrian is small, and further away is the sky, this area can be ignored. The two boundary lines extend upwards from the bottom edge of the image to the distance, with a line height approximately one-quarter of the image height. The image is divided into different areas based on the two boundary lines: the areas on both sides of the vehicle and the driving area in front of the car. Therefore, the pedestrian's location in different areas directly affects their intention to cross the road. When the pedestrian is in M, it means they are crossing the road. When the pedestrian is in L or R, depending on their direction, it may indicate that they are about to cross the road or not. The two boundary lines can be represented in the pixel coordinate system as follows:

[0015]

[0016] As described above, the relative displacement is constituted by the distances from the pedestrian's footpoint to the two boundary lines, denoted as [equation missing]. in and These represent the distances from the foot point to the left and right lines, respectively, and are mathematically expressed as...

[0017]

[0018] In each frame of the observation sequence, the relative displacement of pedestrians needs to be calculated, and these displacements are then connected to form the relative displacement of the observation sequence. From formula (2), it can be seen that the combination of two relative displacements can indicate which region of L, R, and M the traveler is in, i.e., in region L. and All are negative, in the R region. and All are positive, in region M. For the sake of righteousness The fact that the value is negative once again illustrates that relative displacement not only represents the dynamic change in the distance between pedestrians and cars, but also has a strong directional characteristic of pedestrian position. Therefore, relative displacement, as one of the features for estimating pedestrian intentions, enhances the expressive power of features and makes it easier for the network to learn discriminative features.

[0019] Furthermore, in S4, the pedestrian's behavior is not only closely related to their historical trajectory but also highly correlated with their future trajectory. If a pedestrian is in region L at time t, then at some future time t+m, they may move to region M or remain in region L, indicating whether they will cross the road or not. Previous methods for predicting pedestrians' future trajectories are usually based on the pedestrian's historical trajectory sequence FP. obs That is, the learning distribution p(FP) f |FP obs ),in The future trajectory of a pedestrian is represented using the coordinates of their foot points. However, even if multiple pedestrians share the same historical trajectory, they may still follow different paths in the future. This is because a pedestrian's direction often reflects their attention, which directly influences their future trajectory. Therefore, this module adds a pedestrian direction as input, namely, the learned distribution p(FP). f |FP obs OR obs The pedestrian trajectory prediction stream is constructed using a basic Seq2seq LSTM, employing an encoder. tra -Decoder tra The architecture is such that the encoder takes as input a sequence of pedestrian foot coordinates and orientation observed over a time period t, and the decoder outputs the future foot coordinates of the pedestrian from time step t+1 to t+m. Mathematically, the hidden states of the encoder and decoder in the trajectory prediction stream can be represented as follows:

[0020]

[0021]

[0022] During training, the pedestrian trajectory prediction module is trained by minimizing the mean square error (MSE), and the loss function is expressed as follows:

[0023]

[0024] Based on the aforementioned scheme, the pedestrian intention estimation in S5 is a binary classification problem, based on the observed pedestrian direction. Historical relative displacement and future relative displacement Predict the crossing intention of the i-th pedestrian (int∈{0,1}), i.e., learn the distribution p(int|O) obs ,RD obs ,RD fIn this process, the future relative displacement is calculated by the relative displacement generation unit based on the future trajectory. The pedestrian intent estimation stream uses an Encoder-Decoder architecture similar to the trajectory prediction stream, denoted as Encoder. int -Decoder int The difference is that the Encoder int Will receive Encoder int The output of each step is used as input because the Encoder tra The direction of each step is encoded together with the historical trajectory, as this information plays a crucial role in intent estimation. Therefore, it is concatenated with the historical relative displacement as the Encoder. int The input, in addition, in the Decoder int The stage will receive data from the Decoder. tra The output of the Decoder, because the future relative displacement implies the pedestrian's future position and the dynamic changes in the distance between the pedestrian and the car, can better capture the implicit information of the pedestrian's intention. tra With Decoder int A relative displacement generation unit is added between the decoder and the path to convert the future pedestrian trajectory into the future relative displacement. int Receive Encoder int After processing the output of the encoder and a series of future relative displacements, the final output is the classification result of the pedestrian's intent. Mathematically, the hidden states of the encoder and decoder in the intent estimation stream can be represented as follows:

[0025]

[0026]

[0027] The pedestrian intent estimation stream is trained using a binary cross-entropy loss function, defined as follows:

[0028]

[0029] The pedestrian trajectory prediction stream and the intent estimation stream can be jointly trained to achieve end-to-end learning, with the total loss for a single sample sequence being...

[0030] L = L trajectory +L intention (9).

[0031] (III) Beneficial Effects

[0032] Compared with existing technologies, this invention provides a pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM, which has the following advantages:

[0033] 1. In this invention, the prediction of pedestrian trajectories and the estimation of intentions can help autonomous vehicles avoid pedestrians, which is beneficial to safe driving. In addition, understanding pedestrian behavior can also help improve the traffic efficiency of autonomous vehicles, thereby alleviating traffic congestion.

[0034] 2. In this invention, the center point coordinates of the bottom edge of the pedestrian bounding box are used to represent the position of the pedestrian. Compared with the corner points of the bounding box, the center point more vividly represents the position of the pedestrian's feet in contact with the ground, so it is more suitable for representing the position of the pedestrian.

[0035] 3. In this invention, the positional relationship between pedestrians and vehicles is highly correlated. Therefore, relative displacement is proposed to represent the relative position of pedestrians. It can simulate the dynamic changes in distance and position between pedestrians and vehicles, enhance the interaction between pedestrians and vehicles, and further improve the accuracy of pedestrian intention estimation by using the future trajectory of pedestrians. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the overall framework of the pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM proposed in this invention.

[0037] Figure 2 This is a schematic diagram illustrating two representations of pedestrian position in the pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM proposed in this invention.

[0038] Figure 3 This is a schematic diagram of the structure of different regions from the car's perspective in relation to the pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM proposed in this invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example 1

[0041] Reference Figure 1-3 A pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM includes the following steps:

[0042] S1: Overall architecture design, applying regression and classification algorithms to the trajectory and intent respectively to achieve joint training, first building an Encoder. tra -Decoder traThe architecture uses an LSTM to predict the future trajectory of pedestrians based on their direction and historical trajectory. Secondly, a relative displacement generation unit is designed to calculate relative displacement, representing not only the dynamic changes in the distance between the pedestrian and the car but also the pedestrian's relative position information. Finally, an encoder similar to the trajectory prediction unit is used. int -Decoder int LSTM is used to estimate pedestrian intent. It takes pedestrian direction, historical trajectory and relative displacement (historical and future) as input and learns features of similar behaviors to achieve intent estimation.

[0043] S2: Footpoint. In most pedestrian trajectory prediction and intent estimation tasks, the pedestrian's position is typically represented by the top-left corner (x, y) of the bounding box. tl ,y tl ) and bottom right corner (x br ,y br The pixel coordinates of ) are used to represent the bounding_box, which is described as bounding_box = [(x tl ,y tl ),(x br ,y br In addition, some works treat foot coordinates as pedestrian positions, defining them as the midpoints on the bounding box baseline, which can be formally represented as... Two forms of representing pedestrian positions, as follows: Figure 2 As shown in the figure, the foot points are closer to the actual position of a pedestrian's feet, which is more consistent with the actual position of a pedestrian.

[0044] S3: Relative Displacement. The position of a pedestrian in an image, whether at the corner of the bounding box or at the foot, is represented by pixel coordinates. It only represents the absolute position of the pedestrian in the image and is an independent entity, lacking association with contextual information. However, the relative positional relationship between the pedestrian and the car often determines a certain behavior of the pedestrian. For example, when a pedestrian is close to the front of the car, it usually indicates that the pedestrian may be about to cross the road; while when the pedestrian is far away from the vehicle, it may indicate that he is walking away from the vehicle or that he will not cross the road before the car arrives. From the above examples, it can be seen that whether a pedestrian crosses the road is not only related to his direction, but also to his distance from the vehicle. This distance is constantly changing during the movement of the vehicle and the pedestrian. Therefore, it is a relative value between the vehicle and the pedestrian. At the same time, the constant change of this relative value also reflects the change in the relative position between the pedestrian and the vehicle. This change fully affects the pedestrian's trajectory and intention. Therefore, this invention proposes to use relative displacement to represent this relative value in order to simulate the relative positional relationship between the pedestrian and the vehicle.

[0045] S4: Pedestrian trajectory prediction stream. The pedestrian trajectory prediction stream mainly uses the observed historical trajectory and the pedestrian direction at each moment as input. These features are encoded in the encoder, and the decoder of the trajectory prediction stream decodes the encoded information. The output of the decoder at each moment is used as the future trajectory record of the pedestrian.

[0046] S5: Pedestrian Intent Estimation Stream. In this stream, the encoder takes the historical relative displacements of pedestrians and vehicles and the encoder output from the prediction stream as input, and encodes the above features. In the intent estimation stream, the decoder receives the encoded features and the future relative displacement (calculated from the future trajectory output by the decoder in the trajectory prediction stream). The final state of the decoder is the pedestrian intent estimation result.

[0047] In this invention, S3 includes a relative displacement generation unit. Since the pedestrian image is taken from a first-person perspective, the distance between the pedestrian and the vehicle is mainly represented in the image as a lateral distance. Figure 3 Specifically, first, two boundary lines are defined in the image: the left boundary line ll and the right boundary line lr. The width between the two boundary lines can represent the width of the vehicle, or it can represent the area in the image that the car will pass through. Because the upper three-quarters of the image is far from the vehicle and pedestrians are small, and further away is the sky, this part of the image can be ignored. The two boundary lines extend upwards from the bottom edge of the image to the distance, with a line height of approximately one-quarter of the image height. The image is divided into different regions based on the two boundary lines, namely the regions on both sides of the vehicle (…). Figure 3 The L and R areas in the middle) and the driving area in front of the car ( Figure 3 If we consider the M region in the image, then a pedestrian's location within that region directly influences their intention to cross the road. For example, when a pedestrian is in M, it means they are currently crossing the road. However, when a pedestrian is in L or R, depending on their direction, it may indicate that they are about to cross the road or not. These two boundary lines can be represented in pixel coordinates as follows:

[0048]

[0049] As described above, the relative displacement is constituted by the distances from the pedestrian's footpoint to the two boundary lines, denoted as […]. in and These represent the distances from the foot point to the left and right lines, respectively, and are mathematically expressed as...

[0050]

[0051] In each frame of the observation sequence, the relative displacement of pedestrians needs to be calculated, and these displacements are then connected to form the relative displacement of the observation sequence. From formula (2), it can be seen that the combination of two relative displacements can indicate which region of L, R, and M the traveler is in, i.e., in region L. and All are negative, in the R region. and All are positive, in region M. For the sake of righteousness The fact that the value is negative once again illustrates that relative displacement not only represents the dynamic change in the distance between pedestrians and cars, but also has a strong directional characteristic of pedestrian position. Therefore, relative displacement, as one of the features for estimating pedestrian intentions, enhances the expressive power of features and makes it easier for the network to learn discriminative features.

[0052] Example 2

[0053] Reference Figure 1-3 A pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM includes the following steps:

[0054] S1: Overall architecture design, applying regression and classification algorithms to the trajectory and intent respectively to achieve joint training, first building an Encoder. tra -Decoder tra The architecture uses an LSTM to predict the future trajectory of pedestrians based on their direction and historical trajectory. Secondly, a relative displacement generation unit is designed to calculate relative displacement, representing not only the dynamic changes in the distance between the pedestrian and the car but also the pedestrian's relative position information. Finally, an encoder similar to the trajectory prediction unit is used. int -Decoder int LSTM is used to estimate pedestrian intent. It takes pedestrian direction, historical trajectory and relative displacement (historical and future) as input and learns features of similar behaviors to achieve intent estimation.

[0055] S2: Footpoint. In most pedestrian trajectory prediction and intent estimation tasks, the pedestrian's position is typically represented by the top-left corner (x, y) of the bounding box. tl ,y tl ) and bottom right corner (x br ,y br The pixel coordinates of ) are used to represent the bounding_box, which is described as bounding_box = [(x tl ,y tl ),(x br ,y br In addition, some works treat foot coordinates as pedestrian positions, defining them as the midpoints on the bounding box baseline, which can be formally represented as... Two forms of representing pedestrian positions, as follows: Figure 2 As shown in the figure, the foot points are closer to the actual position of a pedestrian's feet, which is more consistent with the actual position of a pedestrian.

[0056] S3: Relative Displacement. The position of a pedestrian in an image, whether at the corner of the bounding box or at the foot, is represented by pixel coordinates. It only represents the absolute position of the pedestrian in the image and is an independent entity, lacking association with contextual information. However, the relative positional relationship between the pedestrian and the car often determines a certain behavior of the pedestrian. For example, when a pedestrian is close to the front of the car, it usually indicates that the pedestrian may be about to cross the road; while when the pedestrian is far away from the vehicle, it may indicate that he is walking away from the vehicle or that he will not cross the road before the car arrives. From the above examples, it can be seen that whether a pedestrian crosses the road is not only related to his direction, but also to his distance from the vehicle. This distance is constantly changing during the movement of the vehicle and the pedestrian. Therefore, it is a relative value between the vehicle and the pedestrian. At the same time, the constant change of this relative value also reflects the change in the relative position between the pedestrian and the vehicle. This change fully affects the pedestrian's trajectory and intention. Therefore, this invention proposes to use relative displacement to represent this relative value in order to simulate the relative positional relationship between the pedestrian and the vehicle.

[0057] S4: Pedestrian trajectory prediction stream. The pedestrian trajectory prediction stream mainly uses the observed historical trajectory and the pedestrian direction at each moment as input. These features are encoded in the encoder, and the decoder of the trajectory prediction stream decodes the encoded information. The output of the decoder at each moment is used as the future trajectory record of the pedestrian.

[0058] S5: Pedestrian Intent Estimation Stream. In this stream, the encoder takes the historical relative displacements of pedestrians and vehicles and the encoder output from the prediction stream as input, and encodes the above features. In the intent estimation stream, the decoder receives the encoded features and the future relative displacement (calculated from the future trajectory output by the decoder in the trajectory prediction stream). The final state of the decoder is the pedestrian intent estimation result.

[0059] In this invention, step S3 includes a relative displacement generation unit. Since the pedestrian image is taken from a first-person perspective, the distance between the pedestrian and the vehicle is mainly represented in the image as a lateral distance. Figure 3 Specifically, first, two boundary lines are defined in the image: the left boundary line ll and the right boundary line lr. The width between the two boundary lines can represent the width of the vehicle, or it can represent the area in the image that the car will pass through. Because the upper three-quarters of the image is far from the vehicle and pedestrians are small, and further away is the sky, this part of the image can be ignored. The two boundary lines extend upwards from the bottom edge of the image to the distance, with a line height of approximately one-quarter of the image height. The image is divided into different regions based on the two boundary lines, namely the regions on both sides of the vehicle (…). Figure 3 The L and R areas in the middle) and the driving area in front of the car ( Figure 3 If we consider the M region in the image, then a pedestrian's location within that region directly influences their intention to cross the road. For example, when a pedestrian is in M, it means they are currently crossing the road. However, when a pedestrian is in L or R, depending on their direction, it may indicate that they are about to cross the road or not. These two boundary lines can be represented in pixel coordinates as follows:

[0060]

[0061] As described above, the relative displacement is constituted by the distances from the pedestrian's footpoint to the two boundary lines, denoted as [equation missing]. in and These represent the distances from the foot point to the left and right lines, respectively, and are mathematically expressed as...

[0062]

[0063] In each frame of the observation sequence, the relative displacement of pedestrians needs to be calculated, and these displacements are then connected to form the relative displacement of the observation sequence. From formula (2), it can be seen that the combination of two relative displacements can indicate which region of L, R, and M the traveler is in, i.e., in region L. and All are negative, in the R region. and All are positive, in region M. For the sake of righteousness The fact that the value is negative once again illustrates that relative displacement not only represents the dynamic change in the distance between pedestrians and cars, but also has a strong directional characteristic of pedestrian position. Therefore, relative displacement, as one of the features for estimating pedestrian intentions, enhances the expressive power of features and makes it easier for the network to learn discriminative features.

[0064] In this invention, the pedestrian's behavior in S4 is not only closely related to the historical trajectory, but also highly related to the future trajectory. For example, if a pedestrian is in region L at time t, then at some future time t+m, they may move to region M or remain in region L, indicating whether they will cross the road or not. Previous methods for predicting the future trajectory of pedestrians are usually based on the pedestrian's historical trajectory sequence FP. obs That is, the learning distribution p(FP) f |FP obs ),in The future trajectory of a pedestrian is represented using the coordinates of their foot points. However, even if multiple pedestrians share the same historical trajectory, they may still follow different paths in the future. This is because a pedestrian's direction often reflects their attention, which directly influences their future trajectory. Therefore, this module adds a pedestrian direction as input, namely, the learned distribution p(FP). f |FP obs OR obs The pedestrian trajectory prediction stream is constructed using a basic Seq2seq LSTM, employing an encoder. tra -Decoder tra The architecture is such that the encoder takes as input a sequence of pedestrian foot coordinates and orientation observed over a time period t, and the decoder outputs the future foot coordinates of the pedestrian from time step t+1 to t+m. Mathematically, the hidden states of the encoder and decoder in the trajectory prediction stream can be represented as follows:

[0065]

[0066]

[0067] During training, the pedestrian trajectory prediction module is trained by minimizing the mean square error (MSE), and the loss function is expressed as follows:

[0068]

[0069] In S5, pedestrian intention estimation is a binary classification problem, based on the observed pedestrian direction. Historical relative displacement and future relative displacement Predict the crossing intention of the i-th pedestrian (int∈{0,1}), i.e., learn the distribution p(int|O) obs ,RD obs ,RD f In this process, the future relative displacement is calculated by the relative displacement generation unit based on the future trajectory. The pedestrian intent estimation stream uses an Encoder-Decoder architecture similar to the trajectory prediction stream, denoted as Encoder. int -Decoder int The difference is that the Encoder int Will receive Encoder int The output of each step is used as input because the Encoder tra The direction of each step is encoded together with the historical trajectory, as this information plays a crucial role in intent estimation. Therefore, it is concatenated with the historical relative displacement as the Encoder. intThe input, in addition, in the Decoder int The stage will receive data from the Decoder. tra The output of the Decoder, because the future relative displacement implies the pedestrian's future position and the dynamic changes in the distance between the pedestrian and the car, can better capture the implicit information of the pedestrian's intention. tra With Decoder int A relative displacement generation unit is added between the decoder and the path to convert the future pedestrian trajectory into the future relative displacement. int Receive Encoder int After processing the output of the encoder and a series of future relative displacements, the final output is the classification result of the pedestrian's intent. Mathematically, the hidden states of the encoder and decoder in the intent estimation stream can be represented as follows:

[0070]

[0071]

[0072] The pedestrian intent estimation stream is trained using a binary cross-entropy loss function, defined as follows:

[0073]

[0074] The pedestrian trajectory prediction stream and the intent estimation stream can be jointly trained to achieve end-to-end learning, with the total loss for a single sample sequence being...

[0075] L = L trajectory +L intention (9).

[0076] In the description herein, it should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0077] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for pedestrian trajectory prediction and pedestrian intention estimation based on dual-stream LSTM, characterized in that, Includes the following steps: S1: Set up a The LSTM architecture uses pedestrian orientation and historical trajectories to predict future pedestrian trajectories, and the pedestrian intent estimation stream uses... Architecture, denoted as ; S2: The top-left pixel coordinates of the pedestrian's bounding box are... and the bottom right pixel coordinates are , described as The foot coordinates are taken as the pedestrian position and defined as the midpoint on the bottom line of the bounding box, denoted as... ; S3: The relative positional relationship between a pedestrian and a car is represented by relative displacement; S4: Pedestrian trajectory prediction stream. This stream uses observed historical trajectories and the pedestrian's direction at each moment as input. The encoder's input is the sequence of pedestrian foot coordinates and direction observed over a period of time, which are encoded. The decoder in the trajectory prediction stream decodes the encoded information, and the decoder's output at each moment serves as the pedestrian's future trajectory. and A relative displacement generation unit is added between them to convert the future pedestrian trajectory into the future relative displacement; S5: Pedestrian Intent Estimation Stream. In the pedestrian intent estimation stream, the encoder takes the historical relative displacement of pedestrians and vehicles and the output of the encoder in the pedestrian intent estimation stream as inputs to complete the feature encoding in the encoder. In the pedestrian intent estimation stream, the decoder receives the encoded features and future relative displacements. The final state of the decoder is the pedestrian intent estimation result.

2. The pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM according to claim 1, characterized in that, S3 includes a relative displacement generation unit, which defines two boundary lines in the image, namely the left boundary line. and the right line The width between the two boundary lines represents the width of the vehicle, or the area in the image that the car will pass through. The relative displacement is defined by the distances from the pedestrian's foot to the two boundary lines, denoted as . ,in and These represent the distances from the foot point to the left and right lines, respectively. The relative displacement of each pedestrian in each frame of the observation sequence needs to be calculated, and these displacements are then connected to form the relative displacement of the observation sequence. .

3. The pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM according to claim 1, characterized in that, The historical trajectory sequence of pedestrians in S4 , , The future trajectory of a pedestrian is represented using the coordinates of the pedestrian's feet. The pedestrian trajectory prediction flow is constructed using a basic Seq2seq LSTM. The architecture, where the encoder input is a time interval. The decoder will output the sequence of pedestrian foot coordinates and orientation observed within the internal data. arrive The future foot coordinates of pedestrians; During training, the pedestrian trajectory prediction module is trained by minimizing the mean square error (MSE).

4. The pedestrian trajectory prediction and intent estimation method based on dual-stream LSTM according to claim 1, characterized in that, In S5, pedestrian intention estimation is a binary classification problem, based on the observed pedestrian direction. Historical relative displacement and future relative displacement Predicting the first The pedestrian's intention to cross the road That is, learning distribution The future relative displacement is calculated by the relative displacement generation unit based on the future trajectory. take over The output of each step is used as input, in The stage will accept from The output, in take over After processing the output and future relative displacement, the final output is the classification result of the pedestrian intention; the pedestrian intention estimation stream is trained using the binary cross-entropy loss function. The pedestrian trajectory prediction stream and the intent estimation stream are jointly trained to achieve end-to-end learning, with the total loss for a single sample sequence being... （9）。

Images