A System and Method for Predicting Pedestrian Trajectories Crossing the Street

Abstract

The present invention discloses a system and method for predicting the trajectory of street-crossing pedestrians, including: performing real-time recognition and detection on the zebra crossing area to obtain the number information of pedestrians and the individual characteristics of pedestrians in this area; obtaining the current position information and historical trajectory of pedestrians; obtaining the signal light information at the zebra crossing position and the position information of vehicles in the waiting area of the zebra crossing; predicting the pedestrian trajectory. The present invention has the characteristics of fast prediction speed, high accuracy, wide applicable scenarios, etc.; by dividing the scene into regions and constructing different prediction logics, the present invention reduces the processing steps, improves the recognition accuracy and speed, and effectively solves the problems of low accuracy and slow speed in predicting the trajectory of street-crossing pedestrians.

Description

Technical Field

[0001] The present invention belongs to the technical field of pedestrian trajectory prediction, and particularly relates to a system and method for predicting the trajectory of pedestrians crossing the street. Background Technique

[0002] In recent years, the decision-making and planning module in autonomous driving technology has always been a research hotspot in the field of vehicle engineering. A good decision-making and planning module needs to be able to accurately understand the driving environment, and monitor and predict the moving trajectories of road participants in real time. As independent, intelligent and vulnerable road users, pedestrians have the ability to make decisions and motion plans independently. Compared with vehicles, the decision-making and motion planning of pedestrians are more uncertain. For example, pedestrians may suddenly accelerate or decelerate irregularly when moving, or even change their motion directions and other behaviors. Therefore, how to accurately predict pedestrian trajectories has always been a difficult point in the decision-making and planning module of autonomous vehicles.

[0003] At present, there are mainly two types of methods for predicting the trajectories of pedestrians crossing the street: namely, prediction methods based on kinematic modeling and prediction methods based on data-driven. The prediction method based on kinematic modeling models the intentions and behaviors of pedestrians by setting constant parameters. Among them, the commonly used pedestrian kinematic models include the constant velocity model, the constant acceleration model and the constant position model. The concept of the social force model is usually added to the prediction of the trajectories of multiple pedestrians. However, the above-mentioned modeling method based on constant parameters is difficult to describe the dynamic change characteristics of the trajectories of pedestrians crossing the street, and the prediction accuracy needs to be further improved. The prediction method based on data-driven usually uses a neural network model to predict the trajectories of pedestrians crossing the street, takes the historical trajectories of pedestrians as the input of the model, and outputs the future trajectories through the trained model. Commonly used neural network models include the Recurrent Neural Network (RNN) model, the Long Short-term Memory (LSTM) neural network model, etc. Its neural network parameters can be obtained by adaptive learning of the model, and it can better cope with the challenges of the dynamic changes of pedestrian movements. However, due to reasons such as a large number of network parameters and complex model structures, it is easy to not converge during training and cannot be effectively applied to all environments.

[0004] Chinese Patent Application No. CN 201910594913.3 discloses a pedestrian trajectory prediction method, which differentially models pedestrian individuals based on a logistic regression model and a social force model, and applies the prediction of the trajectories of pedestrians crossing the street to the decision-making field of autonomous vehicles.

[0005] A method for predicting pedestrian trajectories is disclosed in Chinese Patent Application No. CN201810294015.1. This method combines a long short-term memory network, which is good at dealing with continuous sequence problems in data-driven, with a social affinity map for trajectory prediction.

[0006] The above two patents both belong to the technical category of existing trajectory prediction methods. On the one hand, they are limited by technologies such as modeling methods and parameter selection, and have high requirements for trajectory prediction methods and systems. On the other hand, by training a neural network with pedestrian historical trajectories and predicting future trajectories, due to different network parameters applicable to different scenarios or environments, there are large errors when using the same parameters for prediction, and the accuracy is not high. Summary of the Invention

[0007] Aiming at the deficiencies of the above-mentioned existing technologies, the purpose of the present invention is to provide a system and method for predicting the trajectories of cross-street pedestrians, so as to overcome the problems of low prediction accuracy, slow efficiency, and single applicable scenario existing in the existing cross-street pedestrian trajectory prediction methods. The present invention classifies pedestrian individuals and cross-street environments, and can quickly and accurately predict the trajectories of cross-street pedestrians in different environments.

[0008] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0009] A system for predicting the trajectories of cross-street pedestrians according to the present invention includes: a pedestrian database module, a pedestrian recognition and classification module, a pedestrian trajectory extraction module, a surrounding information collection module, and a pedestrian trajectory prediction module;

[0010] The pedestrian database module is used to store the walking speeds of pedestrians of different genders, different age groups and height ranges, as well as the parameters of LSTM (Long Short-Term Memory Neural Network) and Social LSTM (Social Long Short-Term Memory Neural Network);

[0011] The pedestrian recognition and classification module is used to identify pedestrians within the zebra crossing area and detect their gender, age, height, and mood characteristics;

[0012] The pedestrian trajectory extraction module is used to obtain the current position information and historical trajectories of pedestrians within the zebra crossing area;

[0013] The surrounding information collection module is used to obtain the signal light information at the zebra crossing position and the position information of vehicles in the zebra crossing waiting area;

[0014] The pedestrian trajectory prediction module is used to predict the future trajectories of pedestrians.

[0015] Furthermore, the walking speed is the average value Vm of the walking speeds of the same gender, age group and height range.

[0016] The present invention also provides a method for predicting the trajectory of pedestrians crossing the street, comprising the following steps:

[0017] 1) Identify and detect the zebra crossing area in real time, and obtain the number of pedestrians in this area and the individual characteristics of pedestrians;

[0018] 2) Obtain the current position information and historical trajectory of pedestrians;

[0019] 3) Obtain the signal light information at the zebra crossing position and the position information of vehicles in the waiting area of the zebra crossing;

[0020] 4) Predict the pedestrian trajectory based on the information obtained in the above steps 1)-3), the average walking speed Vm under different genders, age groups and height ranges, the parameters of the long short-term memory neural network, and the parameters of the social long short-term memory neural network.

[0021] Furthermore, the individual characteristics of the pedestrians include: gender, age, height, and mood characteristics.

[0022] Furthermore, the number of pedestrians, gender, age, height, and mood characteristics in step 1) are specifically classified as follows:

[0023] Number of pedestrians: single person, multiple persons;

[0024] Gender: male, female;

[0025] Age: children, teenagers, young people, middle-aged people, elderly people;

[0026] Height: 0-1 meter, 1-1.3 meters, 1.3-1.6 meters, 1.6-1.7 meters, 1.7-1.8 meters, above 1.8 meters;

[0027] Mood characteristics: angry, frightened, happy, calm, sad, surprised.

[0028] Furthermore, in step 2), a multi-object tracking algorithm is used to track the detected pedestrians, and their current position information and historical trajectory are extracted and collected; the current position information and historical trajectory of pedestrians are represented as position coordinates (x, y) on the bird's-eye view of the zebra crossing; among them, the Y-axis on the bird's-eye view coordinate system is defined as the coordinate axis perpendicular to the zebra crossing, the X-axis is defined as the coordinate axis parallel to the zebra crossing, and the coordinate origin (0, 0) is defined as the lower left corner of the bird's-eye view of the zebra crossing.

[0029] Furthermore, the signal light information in step 3) includes two situations: normal green light or flashing green light. By default, pedestrians abide by traffic rules, and there are no pedestrians on the zebra crossing when the signal light at the zebra crossing position is red.

[0030] Furthermore, the specific method of step 4) is as follows:

[0031] 41) The zebra crossing area is divided into Area I and Area II, where Area I accounts for the entire zebra crossing area Area II accounts for the entire zebra crossing area

[0032] 42) Determine the prediction algorithm according to the number of pedestrians in the zebra crossing and the area I or II where the pedestrian to be measured is located. If there are no pedestrians in the zebra crossing, stop the prediction;

[0033] 43) There is one pedestrian in the zebra crossing and the pedestrian to be measured is in Area I:

[0034] For the pedestrian to be measured A, consider the repulsive force of the vehicles in the zebra crossing waiting area on it based on the social force model and the attractive force of the crossing target point on it

[0035] The repulsive force of the vehicle in the zebra crossing waiting area on it is of the magnitude:

[0036]

[0037] In the formula, α1 is the weight coefficient of the influence of the vehicle on the pedestrian, (x1, y1) is the position coordinate of the pedestrian to be measured A, (x2, y2) is the position coordinate of the vehicle in the waiting area. Since the visual range of the pedestrian is limited, the vehicle cannot generate a repulsive force outside the visual range of the pedestrian. The direction of the repulsive force is from the vehicle in the waiting area to the pedestrian to be measured A, and e is the natural constant;

[0038] The attractive force of the crossing target point on it is of the magnitude:

[0039]

[0040] In the formula, β1 and β2 are the weight coefficients of the influence of the crossing target point on the pedestrian, (x1, y1) is the current position coordinate of the pedestrian to be measured A, (x3, y3) is the position coordinate of the target point when the green light is normal. (Since the pedestrian has sufficient time to cross the street at this time, the crossing target point will preferentially select the midpoint of the end of the zebra crossing) The direction of the attractive force is from the pedestrian to be measured A to the crossing target point; (x4, y4) is the position coordinate of the target point when the green light is flashing. (Since the pedestrian has limited time to cross the street at this time, the crossing target point will preferentially select the end of the zebra crossing closest to himself) The direction of the attractive force is from the pedestrian to be measured A to the crossing target point;

[0041] The resultant social force acting on the pedestrian to be measured A is:

[0042]

[0043] The position (x A t+1 , y A t+1 ) of the pedestrian A to be measured after the prediction time Δt is:

[0044]

[0045] Wherein, V A m is the average speed corresponding to the gender, age group and height range of the pedestrian A to be measured, D A is the speed gain coefficient corresponding to the mood characteristic of the pedestrian A to be measured, ω A is the angle between the resultant social force and the X-axis of the zebra crossing, (x A t , y A t ) is the current position coordinate of the pedestrian A to be measured;

[0046] 44) The number of pedestrians inside the zebra crossing is single, and the pedestrian to be measured is located in area II: Use the long short-term neural network (LSTM) optimized by the whale algorithm for prediction;

[0047] 45) The number of pedestrians inside the zebra crossing is multiple, and the pedestrian to be measured is located in area I:

[0048] Define the pedestrian to be measured as B, and consider the repulsive force of the vehicle in the zebra crossing waiting area on it The attractive force of the crossing target point on it The social force of the other pedestrians on it

[0049] The repulsive force of the vehicle in the zebra crossing waiting area on it The magnitude is:

[0050]

[0051] Wherein, α2 and α3 are respectively the weight coefficients of the influence of the vehicle in the waiting area on the pedestrian, (x4, y4) is the current position coordinate of the pedestrian B to be measured, (x5, y5) is the position coordinate of the vehicle in the waiting area, (x n , y n ) are the position coordinates of the other pedestrians, and the direction of the repulsive force is the direction from the vehicle in the waiting area to the pedestrian B to be measured;

[0052] The attractive force of the crossing target point on it The magnitude is:

[0053]

[0054] In the formula, β3 and β4 are the weight coefficients of the influence of the crossing target point on the pedestrian, (x4, y4) is the current position coordinate of the pedestrian, and (x6, y6) is the position coordinate of the target point when the green light is normal. (Since the pedestrian has sufficient time to cross the street at this time, the middle point of the end of the zebra crossing will be preferentially selected as the crossing target point) The direction of the attraction is from the to-be-detected pedestrian B to the crossing target point, (x7, y7) is the position coordinate of the target point when the green light is flashing. (Since the pedestrian has a tight time to cross the street at this time, the end of the zebra crossing closest to himself will be preferentially selected as the crossing target point) The direction of the attraction is from the to-be-detected pedestrian B to the crossing target point;

[0055] The social force of the remaining pedestrians on it has a magnitude of:

[0056]

[0057] In the formula, λ1 and λ2 are the weight coefficients of the influence of the remaining pedestrians on the to-be-predicted pedestrian, (x4, y4) is the current position coordinate of the to-be-detected pedestrian B, and (x P , y P ) is the position coordinate of the remaining pedestrians. (When the green light is normal, pedestrians have sufficient time to cross the street and will preferentially choose to walk in open spaces. The social force of the remaining pedestrians on the to-be-detected pedestrian B is a repulsive force, and the direction of the repulsive force is from the remaining pedestrians to the to-be-detected pedestrian; when the green light is flashing, pedestrians have a tight time to cross the street and lack a sense of security, and will preferentially choose to move in the space where there are other pedestrians. The social force of the remaining pedestrians on the to-be-detected pedestrian B is an attractive force, and the direction of the attractive force is from the to-be-detected pedestrian B to the remaining pedestrians);

[0058] The resultant social force acting on the to-be-detected pedestrian B is:

[0059]

[0060] The position (x B t+1 , y B t+1 ) of pedestrian B after the prediction time Δt is:

[0061]

[0062] In the formula, V B m is the average speed corresponding to the gender, age group and height range of the to-be-detected pedestrian B, D B is the speed gain coefficient corresponding to the mood characteristic of the to-be-detected pedestrian B, ω B is the angle between the resultant social force and the X-axis of the zebra crossing, (x B t , y B t) is the current position coordinate of the pedestrian B to be measured;

[0063] 46) The number of pedestrians within the zebra crossing is multiple, and the pedestrian to be measured is located in Area II:

[0064] Use a social long short-term memory neural network for prediction. The specific prediction process is as follows:

[0065] {(x1 pre ,y1 pre ),(x2 pre ,y2 pre ),…,(x n pre ,y n pre )} = Social LSTM{(x1,y1),(x2,y2),…,(x n ,y n )}

[0066] In the formula, (x1 pre ,y1 pre ),(x2 pre ,y2 pre ),…,(x n pre ,y n pre ) are the predicted position coordinates of all pedestrians 1 to pedestrian n within the area; (x1,y1),(x2,y2),…,(x n ,y n ) are the historical position coordinates of pedestrians 1 to pedestrian n within the area;

[0067] (x1,y1),(x2,y2),…,(x n ,y n ) are the inputs of the social long short-term memory neural network,

[0068] (x1 pre ,y1 pre ),(x2 pre ,y2 pre ),…,(x n pre ,y n pre ) are the outputs of the social long short-term memory neural network.

[0069] Among them, (x n ,y n ) represents the five adjacent historical position coordinates of pedestrian n (x t ,y t ), (x t-1 ,y t-1 ), (x t-2 ,yt-2 ), (x t-3 , y t-3 ), (x t-4 , y t-4 ).

[0070] Further, the specific optimization process of the whale algorithm in step 44) is to use the whale optimization algorithm to optimize three parameters of the long short-term neural network model: the number of iterations, the learning rate, and the number of neurons in the first hidden layer:

[0071] 441) Collect all pedestrian trajectory data in the zebra crossing area randomly within one week;

[0072] 442) Preprocess the pedestrian historical trajectories in the collected pedestrian trajectory data and divide them into a training set, a validation set, and a test set;

[0073] 443) Initialize the number of iterations and the population size of the algorithm, and determine the number of iterations, the learning rate, and the number of neurons in the first hidden layer;

[0074] 444) Randomly generate 10 populations and set the long short-term memory network with the corresponding parameters;

[0075] 445) Use the mean square error between the predicted value and the actual value of the long short-term neural network model as the fitness, calculate the fitness corresponding to each population, take the minimum fitness as the optimal result of this time, and compare it with the global optimal result. If the effect is better, replace it;

[0076] 446) Start iteration and update the three parameters corresponding to the population using the whale algorithm;

[0077] 447) Repeat 445)-446) until the maximum number of iterations;

[0078] 448) Output the number of iterations, the learning rate, and the number of neurons in the hidden layer corresponding to the optimal result.

[0079] Further, the prediction process in step 44) is:

[0080] (x t+1 , y t+1 ) = LSTM 优化 {(x t , y t ), (x t-1 , y t-1 ), (x t-2 , y t-2 ), (x t-3 , y t-3 ), (x t-4 , y t-4 )}

[0081] Wherein, (x t+1 , y t+1 ) is the predicted trajectory of the pedestrian, and (x t , y t ), (x t-1 , y t-1 ), (x t-2 , y t-2 ), (x t-3 , y t-3 ), (x t-4 , y t-4 ) are the historical position coordinates of the pedestrian at five adjacent time steps respectively. The historical position coordinates at five adjacent time steps are used as the input of the long short-term neural network optimized by the whale algorithm to predict the pedestrian position coordinates at the next time step.

[0082] Advantages of the present invention:

[0083] The present invention has the characteristics of fast prediction speed, high accuracy, wide applicable scenarios, etc.; by dividing the scene into regions and constructing different prediction logics, the present invention reduces the processing steps, improves the recognition accuracy and speed, and effectively solves the problems of low accuracy and slow speed in predicting the trajectory of street-crossing pedestrians. Description of the drawings

[0084] Figure 1 is a system framework diagram for predicting the trajectory of street-crossing pedestrians.

[0085] Figure 2 is a schematic diagram of the zebra crossing area division.

[0086] Figure 3 is a prediction scenario diagram when the number of pedestrians in the zebra crossing is one, the pedestrian to be measured is in area I, and the green light is normal.

[0087] Figure 4 is a prediction scenario diagram when the number of pedestrians in the zebra crossing is one, the pedestrian to be measured is in area I, and the green light is flashing.

[0088] Figure 5 is a prediction scenario diagram when the number of pedestrians in the zebra crossing is multiple, the pedestrian to be measured is in area I, and the green light is normal.

[0089] Figure 6 is a prediction scenario diagram when the number of pedestrians in the zebra crossing is multiple, the pedestrian to be measured is in area I, and the green light is flashing. Detailed implementation manners

[0090] For the convenience of understanding by those skilled in the art, the present invention will be further described below in conjunction with the embodiments and the drawings. The content mentioned in the implementation manners does not limit the present invention.

[0091] Refer to Figure 1As shown in the figure, a system for predicting the trajectory of pedestrians crossing the street according to the present invention includes: a pedestrian database module, a pedestrian recognition and classification module, a pedestrian trajectory extraction module, a surrounding information collection module, and a pedestrian trajectory prediction module;

[0092] The pedestrian database module is used to store the walking speeds of pedestrians of different genders, different age groups and height ranges, as well as the parameters of the long short-term memory neural network and the social long short-term memory neural network;

[0093] Among them, the walking speed is the average value Vm of the walking speeds under the same gender, age group and height range.

[0094] The pedestrian recognition and classification module is used to identify pedestrians in the zebra crossing area and detect their gender, age, height, and mood characteristics;

[0095] The pedestrian trajectory extraction module is used to obtain the current position information and historical trajectory of pedestrians in the zebra crossing area;

[0096] The surrounding information collection module is used to obtain the signal light information at the zebra crossing position and the position information of vehicles in the zebra crossing waiting area;

[0097] The pedestrian trajectory prediction module is used to predict the future trajectory of pedestrians.

[0098] The present invention also proposes a method for predicting the trajectory of pedestrians crossing the street, including the following steps:

[0099] 1) Perform real-time recognition and detection on the zebra crossing area to obtain the number of pedestrians in the area and the individual characteristics of pedestrians; the individual characteristics of pedestrians include: gender, age, height, and mood characteristics.

[0100] Specifically, the number of pedestrians, gender, age, height, and mood characteristics in step 1) are specifically divided into:

[0101] Number of pedestrians: single person, multiple people;

[0102] Gender: male, female;

[0103] Age: children, teenagers, young people, middle-aged people, elderly people;

[0104] Height: 0 - 1 meter, 1 - 1.3 meters, 1.3 - 1.6 meters, 1.6 - 1.7 meters, 1.7 - 1.8 meters, over 1.8 meters;

[0105] Mood characteristics: angry, frightened, happy, calm, sad, surprised.

[0106] 2) Obtain the current position information and historical trajectory of pedestrians;

[0107] Specifically, in step 2), the DeepSORT algorithm (multi-object tracking algorithm) is used to track the detected pedestrians, and their current position information and historical trajectories are extracted and collected; the current position information and historical trajectories of the pedestrians are represented as position coordinates (x, y) on the bird's-eye view of the zebra crossing.

[0108] Among them, with reference to Figure 2 As shown, the Y-axis on the bird's-eye view coordinate system is defined as the coordinate axis perpendicular to the zebra crossing, the X-axis is defined as the coordinate axis parallel to the zebra crossing, and the coordinate origin (0, 0) is defined as the lower left corner of the bird's-eye view of the zebra crossing.

[0109] 3) Obtain the signal light information at the zebra crossing position and the position information of the vehicle in the waiting area of the zebra crossing.

[0110] Specifically, the signal light information in step 3) includes two situations: normal green light or flashing green light. By default, pedestrians abide by traffic rules, and when the signal light at the zebra crossing position is red, there are no pedestrians on the zebra crossing.

[0111] 4) Predict the pedestrian trajectory based on the information obtained in steps 1)-3), the average walking speed Vm under different genders, age groups, and height ranges, the parameters of the long short-term memory neural network, and the parameters of the social long short-term memory neural network.

[0112] Specifically, the specific method of step 4) is as follows:

[0113] 41) The zebra crossing area is divided into area Ⅰ and area Ⅱ, where area Ⅰ accounts for area Ⅱ accounts for With reference to Figure 2 As shown;

[0114] 42) Determine the prediction algorithm according to the number of pedestrians in the zebra crossing and whether the pedestrian to be measured is in area Ⅰ or Ⅱ. If there are no pedestrians in the zebra crossing, stop the prediction.

[0115] 43) With reference to Figures 3 to 4 As shown, when the number of pedestrians in the zebra crossing is one and the pedestrian to be measured is in area Ⅰ:

[0116] For pedestrian A to be measured, considering the repulsive force of the vehicle in the zebra crossing waiting area on it based on the social force model the attractive force of the crossing target point on it

[0117] The repulsive force of the vehicle in the zebra crossing waiting area on it The magnitude is:

[0118]

[0119] Wherein, α1 is the weight coefficient of the vehicle's influence on the pedestrian, (x1, y1) is the position coordinate of the to-be-detected pedestrian A, (x2, y2) is the position coordinate of the vehicle in the waiting area. Since the visual range of the pedestrian is limited, the vehicle cannot generate a repulsive force outside the visual range of the pedestrian. The direction of the repulsive force is from the vehicle in the waiting area to the to-be-detected pedestrian A, and e is the natural constant;

[0120] The attraction force of the crossing target point on it is of the magnitude:

[0121]

[0122] Wherein, β1 and β2 are the weight coefficients of the crossing target point's influence on the pedestrian, (x1, y1) is the current position coordinate of the to-be-detected pedestrian A, (x3, y3) is the position coordinate of the target point when the green light is normal (since the pedestrian has sufficient time to cross the street at this time, the middle point of the end of the zebra crossing will be preferentially selected as its crossing target point). The direction of the attraction force is from the to-be-detected pedestrian A to the crossing target point; (x4, y4) is the position coordinate of the target point when the green light is flashing (since the pedestrian has limited time to cross the street at this time, the end of the zebra crossing closest to himself will be preferentially selected as its crossing target point). The direction of the attraction force is from the to-be-detected pedestrian A to the crossing target point;

[0123] The resultant social force acting on the to-be-detected pedestrian A is:

[0124]

[0125] The position (x A t+1 , y A t+1 ) of the to-be-detected pedestrian A after the prediction time Δt is:

[0126]

[0127] Wherein, V A m is the average speed corresponding to the gender, age group and height range of the to-be-detected pedestrian A, D A is the speed gain coefficient corresponding to the mood characteristic of the to-be-detected pedestrian A, ω A is the angle between the resultant social force and the X-axis of the zebra crossing, (x A t , y A t ) is the current position coordinate of the to-be-detected pedestrian A;

[0128] 44) When the number of pedestrians in the zebra crossing is single and the to-be-detected pedestrian is located in area II: The long short-term neural network (LSTM) optimized by the whale algorithm is used for prediction;

[0129] Among them, the specific optimization process of the whale algorithm in step 44) is to use the whale optimization algorithm to optimize three parameters of the long short-term neural network model: the number of iterations, the learning rate, and the number of neurons in the first hidden layer.

[0130] 441) Collect all pedestrian trajectory data in the zebra crossing area randomly within one week.

[0131] 442) Preprocess the pedestrian historical trajectories in the collected pedestrian trajectory data and divide them into a training set, a validation set, and a test set.

[0132] 443) Initialize the number of iterations and the population size of the algorithm, and determine the number of iterations, the learning rate, and the number of neurons in the first hidden layer.

[0133] 444) Randomly generate 10 populations and set the long short-term memory network with the corresponding parameters.

[0134] 445) Take the mean square error between the predicted value and the actual value of the long short-term neural network model as the fitness, calculate the fitness corresponding to each population, take the minimum fitness among them as the optimal result of this time, and compare it with the global optimal result. If the effect is better, replace it.

[0135] 446) Start iteration and update the three parameters corresponding to the population using the whale algorithm.

[0136] 447) Repeat 445)-446) until the maximum number of iterations.

[0137] 448) Output the number of iterations, the learning rate, and the number of neurons in the hidden layer corresponding to the optimal result.

[0138] In the example, the prediction process in step 44) is as follows:

[0139] (x t+1 ,y t+1 ) = LSTM 优化 {(x t ,y t ),(x t-1 ,y t-1 ),(x t-2 ,y t-2 ),(x t-3 ,y t-3 ),(x t-4 ,y t-4 )}

[0140] In the formula, (x t+1 ,y t+1 ) is the predicted trajectory of the pedestrian, (x t ,y t ), (xt-1 , y t-1 ), (x t-2 , y t-2 ), (x t-3 , y t-3 ), (x t-4 , y t-4 ), respectively, are the historical position coordinates of a pedestrian at five adjacent time steps. The historical position coordinates at five adjacent time steps are used as the input of the long short-term neural network optimized by the whale algorithm to predict the position coordinates of the pedestrian at the next time step.

[0141] 45) Refer to Figures 5 to 6 As shown, the number of pedestrians within the zebra crossing is multiple, and the pedestrian to be measured is located in Area I:

[0142] Define the pedestrian to be measured as B. Considering the repulsive force of the vehicles in the zebra crossing waiting area on it based on the social force model The attractive force of the crossing target point on it The social force of the other pedestrians on it

[0143] The repulsive force of the vehicles in the zebra crossing waiting area on it The magnitude is:

[0144]

[0145] In the formula, α2 and α3 are the weight coefficients of the influence of the vehicles in the waiting area on the pedestrians respectively. (x4, y4) is the current position coordinate of the pedestrian to be measured B, (x5, y5) is the position coordinate of the vehicle in the waiting area, (x n , y n ) The position coordinates of the other pedestrians, and the direction of the repulsive force is from the vehicle in the waiting area to the pedestrian to be measured B;

[0146] The attractive force of the crossing target point on it The magnitude is:

[0147]

[0148] In the formula, β3 and β4 are the weight coefficients of the influence of the crossing target point on the pedestrians. (x4, y4) is the current position coordinate of the pedestrian, (x6, y6) is the position coordinate of the target point when the green light is normal, (since the pedestrian has sufficient time to cross the street at this time, the crossing target point will preferentially select the midpoint of the end of the zebra crossing) The direction of the attractive force is from the pedestrian to be measured B to the crossing target point, (x7, y7) is the position coordinate of the target point when the green light is flashing, (since the pedestrian has limited time to cross the street at this time, the crossing target point will preferentially select the end of the zebra crossing closest to himself) The direction of the attractive force is from the pedestrian to be measured B to the crossing target point;

[0149] The social force of the remaining pedestrians on it The magnitude is:

[0150]

[0151] In the formula, λ1 and λ2 are the weight coefficients of the influence of the remaining pedestrians on the pedestrian to be predicted, (x4, y4) is the current position coordinate of the pedestrian to be measured B, (x P , y P ) is the position coordinate of the remaining pedestrians (when the green light is normal, pedestrians have enough time to cross the street and will prefer to walk in open spaces. The social force of the remaining pedestrians on the pedestrian to be measured B is a repulsive force, and the direction of the repulsive force is from the remaining pedestrians to the pedestrian to be measured; when the green light is flashing, pedestrians are short of time to cross the street and lack a sense of security, and will prefer to move in the space where there are other pedestrians. The social force of the remaining pedestrians on the pedestrian to be measured B is an attractive force, and the direction of the attractive force is from the pedestrian to be measured B to the remaining pedestrians);

[0152] The resultant social force acting on the pedestrian to be measured B is:

[0153]

[0154] The position (x B t+1 , y B t+1 ) of pedestrian B after the prediction time Δt is:

[0155]

[0156] In the formula, V B m is the average speed corresponding to the gender, age group and height range of pedestrian B to be measured, D B is the speed gain coefficient corresponding to the mood characteristic of pedestrian B to be measured, ω B is the angle between the resultant social force and the X-axis of the zebra crossing, (x B t , y B t ) is the current position coordinate of pedestrian B to be measured;

[0157] 46) The number of pedestrians inside the zebra crossing is multiple and the pedestrian to be measured is located in area II:

[0158] Use a social long short-term memory neural network for prediction. The specific prediction process is as follows:

[0159] {(x1 pre , y1 pre ), (x2 pre , y2 pre ), …, (x n pre,y n pre )} = Social LSTM{(x1,y1),(x2,y2),…,(x n ,y n )}

[0160] In the formula, (x1 pre ,y1 pre ), (x2 pre ,y2 pre ), …, (x n pre ,y n pre ) are the predicted position coordinates of all pedestrians 1 to pedestrian n in the area; (x1,y1), (x2,y2), …, (x n ,y n ) are the historical position coordinates of pedestrians 1 to pedestrian n in the area;

[0161] (x1,y1), (x2,y2), …, (x n ,y n ) are the inputs of the social long short-term memory neural network,

[0162] (x1 pre ,y1 pre ), (x2 pre ,y2 pre ), …, (x n pre ,y n pre ) are the outputs of the social long short-term memory neural network.

[0163] Among them, (x n ,y n ) represents the five adjacent historical position coordinates of pedestrian n (x t ,y t ), (x t-1 ,y t-1 ), (x t-2 ,y t-2 ), (x t-3 ,y t-3 ), (x t-4 ,y t-4 ).

[0164] There are many specific application ways for the present invention. The above description is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art in this technical field, several improvements can be made without departing from the principle of the present invention, and these improvements should also be regarded as the protection scope of the present invention.

Claims

1. A method for predicting the trajectory of a pedestrian crossing the street, characterized in that, It includes the following steps: 1) Real-time identify and detect the zebra crossing area, and obtain the number of pedestrians in this area and the individual characteristics of pedestrians; 2) Obtain the current position information and historical trajectory of pedestrians; 3) Obtain the signal light information at the zebra crossing position and the position information of vehicles in the waiting area of the zebra crossing; 4) Predict the pedestrian trajectory according to the information obtained in the above steps 1)-3), the average walking speed Vm under different genders, age groups and height ranges, the parameters of the long short-term memory neural network, and the parameters of the social long short-term memory neural network; In step 2), a multi-target tracking algorithm is used to track the detected pedestrians, and their current position information and historical trajectory are extracted and collected; the current position information and historical trajectory of pedestrians are represented as position coordinates (x, y) on the bird's-eye view of the zebra crossing; among them, the Y-axis on the bird's-eye view coordinate system is defined as the coordinate axis perpendicular to the zebra crossing, the X-axis is defined as the coordinate axis parallel to the zebra crossing, and the coordinate origin (0, 0) is defined as the lower left corner of the bird's-eye view of the zebra crossing; The specific method of step 4) is as follows: 41) The zebra crossing area is divided into Area I and Area II, where Area I accounts for the entire zebra crossing area Area II accounts for the entire zebra crossing area 42) Determine the prediction algorithm according to the number of pedestrians in the zebra crossing and the area Ⅰ or Ⅱ where the pedestrian to be measured is located. If there is no pedestrian in the zebra crossing, stop the prediction; 43) The number of pedestrians in the zebra crossing is one, and the pedestrian to be measured is located in area Ⅰ: Pedestrian A to be measured, considering the repulsive force of vehicles in the zebra crossing waiting area on it based on the social force model The attractive force of the crossing target point on it The repulsive force of the vehicle on the zebra crossing waiting area The magnitude is: Wherein, α1 is the weight coefficient of the vehicle's influence on the pedestrian, (x1, y1) is the position coordinate of the pedestrian A to be measured, (x2, y2) is the position coordinate of the vehicle in the waiting area, and the direction of the repulsive force is from the vehicle in the waiting area to the pedestrian A to be measured, and e is the natural constant; The attraction of the cross-street target point to it The magnitude is: Wherein, β1 and β2 are the weight coefficients of the influence of the cross-street target point on the pedestrian, (x1, y1) is the current position coordinate of the to-be-detected pedestrian A, (x3, y3) is the position coordinate of the target point when the green light is normal, (x4, y4) is the position coordinate of the target point when the green light is flashing, and the direction of the attraction is from the to-be-detected pedestrian A to the cross-street target point; The resultant social force acting on the pedestrian to be measured A is: The position (x A t +1 , y A t +1 ) of pedestrian A to be measured after the prediction time Δt is as follows: where, V A m is the average speed corresponding to the gender, age group and height range of the pedestrian A to be measured, D A is the speed gain coefficient corresponding to the mood characteristic of the pedestrian A to be measured, ω A is the angle between the resultant social force and the X-axis of the zebra crossing, (x A t , y A t ) is the current position coordinate of the pedestrian A to be measured; 44) The number of pedestrians in the zebra crossing is one, and the pedestrian to be measured is located in area Ⅱ: Use the long short-term memory neural network optimized by the whale algorithm for prediction; 45) The number of pedestrians in the zebra crossing is multiple, and the pedestrian to be measured is located in area Ⅰ: Define the pedestrian to be measured as B, and consider the repulsive force of the vehicles in the zebra crossing waiting area on it based on the social force model The attractive force of the crossing target point on it The social force of other pedestrians on it The repulsive force of the vehicle in the zebra crossing waiting area The magnitude is: Wherein, α2 and α3 are respectively the weight coefficients of the influence of the vehicles in the waiting area on pedestrians, (x'4, y'4) is the current position coordinate of the to-be-detected pedestrian B, (x5, y5) is the position coordinate of the vehicle in the waiting area, (x n , y n ) are the position coordinates of the remaining pedestrians, and the direction of the repulsive force is from the vehicle in the waiting area to the to-be-detected pedestrian B; The attraction of the cross-street target point to it The magnitude is: In the formula, β3 and β4 are the weight coefficients of the influence of the cross-street target point on the pedestrian, (x'4, y'4) is the current position coordinate of the pedestrian, (x6, y6) is the position coordinate of the target point when the green light is normal, (x7, y7) is the position coordinate of the target point when the green light is flashing, and the direction of the attraction is from the to-be-detected pedestrian B to the cross-street target point; The social force of the remaining pedestrians The magnitude is: where λ1 and λ2 are the weight coefficients of the influence of other pedestrians on the pedestrian to be predicted, (x'4, y'4) is the current position coordinate of the pedestrian B to be measured, and (x P , y P ) are the position coordinates of other pedestrians; The resultant social force acting on the pedestrian to be measured B is: The position (x B t +1 , y B t +1 ) of pedestrian B after the prediction time Δt is as follows: wherein, V B m is the average speed corresponding to the gender, age group and height range of the pedestrian B to be measured, D B is the speed gain coefficient corresponding to the mood characteristics of the pedestrian B to be measured, ω B is the angle between the resultant social force and the X-axis of the zebra crossing, (x B t , y B t ) is the current position coordinate of the pedestrian B to be measured; 46) The number of pedestrians in the zebra crossing is multiple, and the pedestrian to be measured is located in area Ⅱ: Use the social long short-term memory neural network for prediction. The specific prediction process is as follows: {(x1 pre ,y1 pre ),(x2 pre ,y2 pre ),…,(x n pre ,y n pre )} = Social LSTM{(x1,y1),(x2,y2),…,(x n ,y n )} where (x1 pre , y1 pre ), (x2 pre , y2 pre ), …, (x n pre , y n pre ) are the predicted position coordinates of all pedestrians 1 to pedestrian n in the area; (x1, y1), (x2, y2), …, (x n , y n ) are the historical position coordinates of pedestrians 1 to pedestrian n in the area; (x1, y1), (x2, y2), …, (x n , y n ) are the inputs of the social long short-term memory neural network, and (x1 pre , y1 pre ), (x2 pre , y2 pre ), …, (x n pre , y n pre ) are the outputs of the social long short-term memory neural network.

2. The prediction method of the pedestrian trajectory across the street according to claim 1, characterized in that, The individual characteristics of the pedestrians include: gender, age, height, and mood characteristics.

3. The prediction method of the trajectory of pedestrians crossing the street according to claim 1, wherein The specific optimization process of the whale algorithm in step 44) is to use the whale optimization algorithm to optimize three parameters of the long short-term memory neural network model: the number of iterations, the learning rate, and the number of neurons in the first hidden layer: 441) Collect all pedestrian trajectory data in the zebra crossing area randomly within a week; 442) Preprocess the historical trajectories of pedestrians in the collected pedestrian trajectory data, and divide them into a training set, a validation set, and a test set; 443) Initialize the number of iterations and the population size of the algorithm, and determine the number of iterations, the learning rate, and the number of neurons in the first hidden layer; 444) Randomly generate 10 populations, and set the long short-term memory neural network with the corresponding parameters; 445) Take the mean square error between the predicted value and the actual value of the long short-term memory neural network model as the fitness, calculate the fitness corresponding to each population, take the minimum fitness among them as the optimal result of this time, and compare it with the global optimal result. If the effect is better, replace it; 446) Start iteration, and use the whale algorithm to update the three parameters corresponding to the population; 447) Repeat 445)-446) until the maximum number of iterations; 448) Output the number of iterations, the learning rate, and the number of neurons in the hidden layer corresponding to the optimal result.

4. The prediction method of the trajectory of street-crossing pedestrians according to claim 1, characterized in that The prediction process in step 44) is as follows: (xt +1 ,yt +1 ) = LSTM 优化 {(x t ,y t ),(x t-1 ,y t-1 ),(x t-2 ,y t-2 ),(x t-3 ,y t-3 ),(x t-4 ,y t-4 )} where, (x t+1 , y t+1 ) is the predicted trajectory of the pedestrian, and (x t , y t ), (x t-1 , y t-1 ), (x t-2 , y t-2 ), (x t-3 , y t-3 ), (x t-4 , y t-4 ) are the historical position coordinates of the pedestrian at five adjacent time steps respectively. The historical position coordinates at five adjacent time steps are used as the input of the long short-term memory neural network optimized by the whale algorithm to predict the pedestrian position coordinates at the next time step.

5. A system for predicting the trajectory of pedestrians crossing the street, based on the method described in claim 1, characterized in that, The system includes: a pedestrian database module, a pedestrian recognition and classification module, a pedestrian trajectory extraction module, a surrounding information collection module, and a pedestrian trajectory prediction module; The pedestrian database module is used to store the walking speeds of pedestrians of different genders, different age groups, and height ranges, as well as the parameters of long short-term memory neural networks and social long short-term memory neural networks; The pedestrian recognition and classification module is used to identify pedestrians within the zebra crossing area and detect their individual characteristics; The pedestrian trajectory extraction module is used to obtain the current position information and historical trajectories of pedestrians within the zebra crossing area; The surrounding information collection module is used to obtain the signal light information at the zebra crossing position and the position information of vehicles in the zebra crossing waiting area; The pedestrian trajectory prediction module is used to predict the future trajectories of pedestrians.

Images