Pedestrian trajectory prediction method, vehicle control method, device, equipment and medium

Abstract

The application discloses a pedestrian trajectory prediction method, a vehicle control method, a device, equipment and a medium, and relates to the field of automatic driving, and comprises the following steps: acquiring pedestrian data, vehicle data and environment data collected by a camera, and removing irregular trajectory data in the pedestrian data based on a preset data processing mechanism; calculating the pedestrian data, the vehicle data and the environment data after removal according to a target classification algorithm to predict a target position of a target pedestrian; and calculating the target position by using target position attraction, repulsion between pedestrians and environmental repulsion determined based on the result output by a target model to predict a target trajectory of the target pedestrian to the target position. It can be seen that the application predicts the target position of the pedestrian based on the pedestrian data after removal, improves the prediction accuracy, and introduces the target position attraction, the repulsion between pedestrians and the environmental repulsion, reduces the possibility of crossing between pedestrians and pedestrians or between pedestrians and obstacles, and makes the predicted trajectory more realistic.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a pedestrian trajectory prediction method, vehicle control method, device, equipment and medium. Background Technology

[0002] Pedestrian trajectory prediction is a key issue of concern for many companies and scholars. In the autonomous driving industry, the accuracy of pedestrian trajectory prediction greatly affects the safety and comfort of vehicle driving.

[0003] Unlike pedestrian trajectory prediction in general scenarios, pedestrian trajectories at traffic light intersections have distinct characteristics, primarily including changes with traffic light transitions and strong destination attributes. However, this doesn't mean that pedestrian trajectory prediction in this scenario is simpler. First, traffic lights cannot be used alone as the standard for whether pedestrians cross the intersection, as not all trajectories can be guaranteed to comply with traffic regulations. Second, some road sections have waiting areas at the midpoint of the road, where some pedestrians might choose to wait. This is unfavorable for trajectory prediction methods using neural network models, as numerically, the trajectory appears to suddenly stop. Furthermore, some pedestrians might be about to cross the intersection, but the green light ends, forcing them back to the waiting area. Predicting such trajectories is nearly impossible for neural network models, firstly because this is a low-probability event, and secondly because neural networks, which require large amounts of data to learn, cannot acquire this feature, and thirdly, the reverse trajectory is numerically non-differentiable. Therefore, traditional methods for predicting pedestrian trajectories at traffic light intersections are ineffective.

[0004] Therefore, how to improve the prediction accuracy of pedestrian trajectories at traffic light intersections in order to improve driving smoothness and comfort is an urgent problem to be solved in this field. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a pedestrian trajectory prediction method, vehicle control method, device, equipment, and medium, which can improve the prediction accuracy of pedestrian trajectories at traffic light intersections, thereby improving driving smoothness and comfort. The specific solution is as follows:

[0006] Firstly, this application discloses a method for predicting pedestrian trajectories, including:

[0007] Acquire pedestrian data, vehicle data, and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal;

[0008] The target location of the target pedestrian is predicted by calculating the removed pedestrian data, vehicle data, and environmental data based on the target classification algorithm.

[0009] The target location is calculated using the attraction, repulsion between pedestrians, and environmental repulsion determined by the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target location.

[0010] Optionally, the step of removing irregular trajectory data from the pedestrian data based on a preset data processing mechanism includes:

[0011] The predicted trajectory data of the target pedestrian in the previous frame and the actual trajectory data in the current frame are determined from the pedestrian data.

[0012] Determine a first difference between the actual trajectory data of the current frame and the predicted trajectory data of the previous frame, and determine a first target value corresponding to the first difference;

[0013] If the absolute value of the first target value is greater than the first preset threshold, then the trajectory data of the target pedestrian before the actual trajectory data of the current frame will be removed.

[0014] And / or, the step of removing irregular pedestrian trajectory data from the pedestrian data based on a preset data processing mechanism includes:

[0015] The actual trajectory data of the target pedestrian in the previous frame and the actual trajectory data in the current frame are determined from the pedestrian data.

[0016] Determine a second difference between the actual trajectory data of the current frame and the actual trajectory data of the previous frame, and determine a second target value corresponding to the second difference;

[0017] If the absolute value of the second target value is greater than the second preset threshold, then the trajectory data of the target pedestrian before the actual trajectory data of the current frame will be removed.

[0018] Optionally, determining the first target value corresponding to the first difference includes:

[0019] The product of the first difference and the first target scaling ratio is determined as the first target value corresponding to the first difference;

[0020] Accordingly, determining the second target value corresponding to the second difference includes:

[0021] The product of the second difference and the second target scaling ratio is determined as the second target value corresponding to the second difference.

[0022] Optionally, the step of calculating the target location of the target pedestrian based on the removed pedestrian data, the vehicle data, and the environmental data according to the target classification algorithm includes:

[0023] Determine the feature vectors corresponding to the removed pedestrian data, vehicle data, and environmental data, and combine the feature vectors to obtain several new feature vectors;

[0024] The new feature vectors are input into the corresponding base predictors to obtain several prediction probabilities.

[0025] The total probability of the target is determined based on the plurality of predicted probabilities and the weight value of each predicted probability, and the target position is determined based on the total probability of the target.

[0026] Optionally, before calculating the target location using the target location attraction, inter-pedestrian repulsion, and environmental repulsion determined based on the target model output, the method further includes:

[0027] A first neural network corresponding to the attractive force at the target location, a second neural network corresponding to the repulsive force between pedestrians, and a third neural network corresponding to the repulsive force of the environment are determined.

[0028] The target model is constructed based on the first neural network, the second neural network, and the third neural network.

[0029] Optionally, the step of calculating the target location using the target location attraction, inter-pedestrian repulsion, and environmental repulsion determined based on the target model output, to predict the target trajectory of the target pedestrian reaching the target location, includes:

[0030] Based on the target time output by the first neural network in the target model, the attractiveness of the target location is calculated; wherein, the target time is the time it takes for the target pedestrian to reach the target location from the current location when there are no obstacles.

[0031] The repulsive force between pedestrians is calculated based on the target collision coefficient output by the second neural network in the target model.

[0032] The environmental repulsion force is calculated based on the target environment coefficient output by the third neural network in the target model;

[0033] The target location is calculated using the attraction force of the target location, the repulsion force between pedestrians, and the repulsion force of the environment to predict the target trajectory of the target pedestrian to reach the target location.

[0034] Secondly, this application discloses a vehicle control method, including:

[0035] Acquire pedestrian data, vehicle data, and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal;

[0036] The target location of the target pedestrian is predicted by calculating the removed pedestrian data, vehicle data, and environmental data based on the target classification algorithm.

[0037] The target location is calculated using the attraction, repulsion between pedestrians, and environmental repulsion determined by the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target location;

[0038] The vehicle is controlled based on the pedestrian data, the vehicle data, the environmental data, and the target trajectory.

[0039] Thirdly, this application discloses a pedestrian trajectory prediction device, comprising:

[0040] The data removal module is used to acquire pedestrian data, vehicle data and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal.

[0041] The target location determination module is used to calculate the removed pedestrian data, vehicle data and environmental data according to the target classification algorithm to predict the target location of the target pedestrian;

[0042] The target trajectory determination module is used to calculate the target position using the target position attraction, inter-pedestrian repulsion and environmental repulsion determined by the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target position.

[0043] Fourthly, this application discloses a vehicle control device, comprising:

[0044] The data removal module is used to acquire pedestrian data, vehicle data and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal.

[0045] The target location determination module is used to calculate the removed pedestrian data, vehicle data and environmental data according to the target classification algorithm to predict the target location of the target pedestrian;

[0046] The target trajectory determination module is used to calculate the target position using the target position attraction, inter-pedestrian repulsion and environmental repulsion determined based on the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target position;

[0047] The vehicle control module is used to control the vehicle based on the pedestrian data, the vehicle data, the environmental data, and the target trajectory.

[0048] Fifthly, this application discloses an electronic device, comprising:

[0049] Memory, used to store computer programs;

[0050] A processor for executing the computer program to implement the aforementioned disclosed method.

[0051] Sixthly, this application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method.

[0052] As can be seen, this application proposes a pedestrian trajectory prediction method, including: acquiring pedestrian data, vehicle data, and environmental data collected by a camera, and removing irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal; calculating the target position of the target pedestrian based on the removed pedestrian data, the vehicle data, and the environmental data according to a target classification algorithm; and calculating the target position using the target position attraction, inter-pedestrian repulsion, and environmental repulsion determined based on the target model output to predict the target trajectory of the target pedestrian to the target position. In summary, this application introduces a preset data processing mechanism to remove irregular trajectory data from the pedestrian data, and then predicts the target position of the pedestrian based on the removed pedestrian data, thereby improving the prediction accuracy of the target position. On the other hand, the target model in this application transforms various influencing factors in a complex environment into the expression of "forces," using forces with a certain magnitude and direction to describe the constraints on the target pedestrian, thereby calculating the target trajectory of the target pedestrian in that environment. It is understood that the aforementioned various influencing factors include other pedestrians or obstacles, etc. In this way, this application reduces the possibility of the target pedestrian intersecting with other pedestrians or obstacles in the environment, making the predicted target trajectory more realistic and accurate. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 This is a flowchart of a pedestrian trajectory prediction method disclosed in this application;

[0055] Figure 2 This is a schematic diagram of the structure of a Catboost model disclosed in this application;

[0056] Figure 3 This is a schematic diagram of the structure of a target model disclosed in this application.

[0057] Figure 4 This is a schematic diagram of a first neural network disclosed in this application;

[0058] Figure 5 This is a schematic diagram of a second neural network disclosed in this application;

[0059] Figure 6 This is a flowchart of a vehicle control method disclosed in this application;

[0060] Figure 7 This is a flowchart of a specific vehicle control method disclosed in this application;

[0061] Figure 8 This is a schematic diagram of a scenario for controlling a vehicle based on a vehicle control method disclosed in this application;

[0062] Figure 9 This is a schematic diagram of the pedestrian trajectory prediction device disclosed in this application;

[0063] Figure 10 This is a schematic diagram of the structure of a vehicle control device disclosed in this application;

[0064] Figure 11 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] When predicting pedestrian trajectories at traffic light intersections, traffic lights alone cannot be used as the sole criterion for whether a pedestrian should cross the intersection. This is because it cannot be guaranteed that all trajectories will comply with traffic regulations. Secondly, some pedestrians may choose to wait in the designated waiting area at the midpoint of the road, which is unfavorable for trajectory prediction methods using neural network models. Furthermore, some pedestrians may turn back, which is a low-probability event. Neural networks, which require a large amount of data to learn, cannot learn this feature, and the reverse trajectory is not numerically differentiable. Therefore, traditional methods for predicting pedestrian trajectories at traffic light intersections are ineffective.

[0067] Therefore, this application proposes a pedestrian trajectory prediction scheme and a vehicle control scheme, which can improve the prediction accuracy of pedestrian trajectories at traffic light intersections, thereby improving driving smoothness and comfort.

[0068] This application discloses a method for predicting pedestrian trajectories. (See also...) Figure 1 As shown, the method includes:

[0069] Step S11: Acquire pedestrian data, vehicle data and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal.

[0070] In this embodiment, the pedestrian data includes data such as speed, position, acceleration, and orientation of pedestrians within the detection range of the intelligent forward-looking camera; the vehicle data includes data such as speed, position, acceleration, and orientation of motor vehicles and non-motor vehicles (bicycles, electric vehicles, etc.) within the detection range of the intelligent forward-looking camera; and the environmental data includes data such as curbs, road attributes (width, road alignment), zebra crossings, traffic lights, obstacles, and road shadows within the detection range of the intelligent forward-looking camera.

[0071] It is understandable that some pedestrians may choose to wait in the waiting area when crossing an intersection. This behavior is numerically represented by a sudden stop in the trajectory. In this case, using a neural network model to predict the trajectory is inaccurate. Furthermore, some pedestrians may turn back, and the reverse trajectory is not numerically differentiable. Therefore, using traditional methods to predict the trajectory is not effective.

[0072] Therefore, this embodiment uses a preset data processing mechanism to remove the irregular trajectory data to obtain pedestrian data after removal. Furthermore, trajectory prediction is performed based on the pedestrian data after removal to improve the accuracy of trajectory prediction.

[0073] In one specific implementation, the previous frame predicted trajectory data D of the target pedestrian is determined from the pedestrian data. t-1,pred and the actual trajectory data D of the current frame tWherein, the predicted trajectory data of the previous frame is the trajectory data of the current frame predicted in the previous frame. And a first difference D between the actual trajectory data of the current frame and the predicted trajectory data of the previous frame is determined. t -D t-1,pred Then scale according to the first target ratio. The first difference is scaled to the range of 0-1 to obtain the first target value. Furthermore, the absolute value of the first target value is compared with a first preset threshold. If the absolute value of the first target value is greater than the first preset threshold, then the trajectory data before the actual trajectory data of the current frame is removed.

[0074] For example, if the predicted actual trajectory data of the current frame is 2 meters based on the pedestrian's current movement trend, but the target pedestrian suddenly stops moving in the current frame, then the actual trajectory data of the current frame is 0 meters. Therefore, the first difference is -2 meters. Assuming the first target scaling ratio is 1 / 10 and the first preset threshold is 0.1, then the first target value calculated based on this is -1 / 5. The sign of the target value can indicate the direction of movement. It can be seen that the absolute value of the first target value is greater than the first preset threshold, indicating that the trajectory data before the actual trajectory data of the current frame is inaccurate. Therefore, the trajectory data before the actual trajectory data of the current frame is removed accordingly.

[0075] For example, if the target pedestrian is stationary in the previous few frames, the predicted actual trajectory data for the current frame is 0 meters based on this movement trend. However, if the target pedestrian suddenly accelerates in the current frame, the actual trajectory data for the current frame will be 1 meter. Therefore, the first difference is 1 meter. Assuming the first target scaling ratio is 1 / 10 and the first preset threshold is 0.1, the first target value calculated based on this is 0.1. It can be seen that the absolute value of the first target value is equal to the first preset threshold, indicating that the trajectory data before the actual trajectory data of the current frame is accurate, so there is no need to remove the trajectory data before the actual trajectory data of the current frame.

[0076] For example, if the predicted actual trajectory data of the current frame is 2.5 meters based on the pedestrian's current movement trend, and the actual trajectory data of the pedestrian in the current frame is 2 meters, then the first difference is 0.5 meters. Assuming the first target scaling ratio is 1 / 10 and the first preset threshold is 0.1, then the first target value calculated based on this is 0.05. It can be seen that the absolute value of the first target value is less than the first preset threshold, indicating that the trajectory data before the actual trajectory data of the current frame is accurate, so there is no need to remove the trajectory data before the actual trajectory data of the current frame.

[0077] In another specific implementation, the previous frame's actual trajectory data D of the target pedestrian is determined from the pedestrian data. t-1 and the actual trajectory data D of the current frame tAnd determine the second difference D between the actual trajectory data of the current frame and the actual trajectory data of the previous frame. t -D t-1 Then scale according to the second target. The second difference is scaled to the range of 0-1 to obtain the second target value. Furthermore, the second target value is compared with the second preset threshold. If the second target value is greater than the second preset threshold, the trajectory data preceding the actual trajectory data of the current frame is removed. The first and second preset thresholds can be set according to the actual engineering project and business scenario. Additionally, the first and second target scaling ratios are used to scale the first difference and the second difference, and their specific values ​​can also be set according to actual needs.

[0078] For example, taking the current direction of the pedestrian's forward movement as the positive direction, if the pedestrian walks 1 meter forward in the previous frame and then turns back 2 meters in the current frame, the second difference is -3 meters. Assuming the second target scaling ratio is 1 / 10 and the first preset threshold is 0.1, the calculated second target value is -3 / 10. It can be seen that the absolute value of the second target value is greater than the second preset threshold. Furthermore, due to the turning back, the trajectory data before the actual trajectory data in the current frame is non-differentiable. Since the subsequent physical neural model algorithm cannot handle non-differentiable data, the trajectory data before the actual trajectory data in the current frame is removed accordingly.

[0079] Step S12: Calculate the removed pedestrian data, vehicle data, and environmental data according to the target classification algorithm to predict the target location of the target pedestrian.

[0080] Figure 2 This is a schematic diagram of the Catboost (Categorical + Boosting) model used in the target classification algorithm. First, the target feature vector D = (X...) is determined. k ,y k ) k＝1,2,...n ,in, It is a feature vector corresponding to the removed pedestrian data, vehicle data, and environmental data. Specifically, the removed pedestrian data, vehicle data, and environmental data collected at a certain point in time contain n features, namely x 1 x 2 …x n A feature vector is formed by combining the n features corresponding to k time points (k = 1, 2, ... k). therefore, It is the first feature collected at the k-th time point. The nth feature is collected at the kth time point, y k The feature vector is the label vector. Further, the feature vector is processed using target statistical methods, and then combined based on the feature combination function to obtain n new feature vectors. During the combination process, CatBoost can combine all categorical features and use combined features with high intrinsic correlation as new features for modeling. For example, in trajectory prediction, the speed of pedestrians can be combined with the speed of other pedestrians or the speed of surrounding pedestrians. Further, the n new feature vectors are input into the corresponding n base predictors to obtain n prediction probabilities. It should be noted that the process of obtaining prediction probabilities by the base predictors has been described in related technologies and will not be elaborated here. Finally, the total target probability P is determined based on the n prediction probabilities and the weight value of each prediction probability, and the target position is determined based on the total target probability P, P = a1P1 + a2P2 + ... + a n P n , where a n P is the weight value for each predicted probability. n This is the predicted probability output by the nth base predictor. Figure 2 In this algorithm, balls with the same number represent the same feature, and a larger ball indicates a larger weight for that feature. It should be noted that the target classification algorithm can transform the input data into corresponding category information. For example, assuming there is a construction area in front of a pedestrian, this embodiment can determine the detailed information of the construction area, such as its category information, through the target classification algorithm. It can then directly use the category information of the construction area and other data as features to obtain the target location the pedestrian will reach after avoiding the construction area. However, some projects use Long Short-Term Memory (LSTM) networks to obtain the target location. It should be noted that LSTM networks cannot transform input data into corresponding category information, and because obstacle environment data, such as construction areas, is relatively scarce in all environmental data (because construction areas are less common during normal walking), LSTM networks cannot achieve good location prediction based on limited environmental data. In other words, for the same data, the target classification algorithm in this embodiment can more accurately predict the target location.

[0081] Step S13: Calculate the target location using the target location attraction, inter-pedestrian repulsion, and environmental repulsion determined by the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target location.

[0082] In this embodiment, the target model is a PNM (Physical Neural Model). The target model is explained below: at any time t, the position of the i-th person in the crowd is represented as... The coordinates of a pedestrian entering and exiting the data collection range of the intelligent forward-looking camera can be represented as {q} 0 ,q 1 ,q 2 ,...q T The set of}, where Let be the speed of pedestrian i at time t. Simultaneously, at time t, the intelligent forward-looking camera collects data on the surrounding crowd of person i, represented as... Here, k represents the surrounding pedestrians.

[0083] Furthermore, the walking trajectory of the i-th pedestrian in the crowd can be represented as: Where, θ and These are learnable parameters; E represents environment variables; θ contains interpretable parameters. It contains uninterpretable parameters (such as neural network weights); the dynamics of the pedestrian (i.e., the agent) are controlled by a function f, which depends on time t, the trajectory q(t) of the pedestrian to be predicted, and the target position q of the pedestrian to be predicted. T And Π(t), where Π(t) represents the summation over time, i.e., Π(t) = 0 + 1 + ... + t. Given the target classification algorithm with respect to q... T After that, you can get Since this application pre-processes some irregular trajectory data based on a preset data processing mechanism, it is assumed that the pedestrian's position and velocity are first and second order differentiable, thus yielding... Where p(t) is the pedestrian's position at time t. It is the speed of the pedestrian at time t. It is the acceleration of the pedestrian at time t, which can be represented by a kinematic model, and the equation of the kinematic model is: F goal For the target location's attractiveness, F collision For the repulsive force between pedestrians, F enviorment As environmental repulsive forces, all three forces mentioned above are implemented through neural networks; see [link / reference]. Figure 3 As shown, Figure 3 The flowchart for trajectory prediction based on the target model shows that the target model includes a first neural network corresponding to the attraction of the target location, a second neural network corresponding to the repulsion between pedestrians, and a third neural network corresponding to the repulsion of the environment.

[0084] Figure 4For the first neural network corresponding to the attraction of the target location, the first neural network considers the speed and position of the target pedestrian, i.e., q t The data is encoded and then fed into a Long Short-Term Memory (LSTM) network to capture temporal features. After transformation through a linear layer, the output is a function F. goal The first result at time t Furthermore, the target location is processed by a multilayer perceptron (MLP), and the output is a function F. goal Second result at time 0-T Among them, T=Δt1+Δt2+...+Δt n Δt1 represents a short time interval, for example, dividing 1 second into n segments, with the first segment representing Δt1. Further, the XOR operation result of the first and second results is passed through a multilayer perceptron to output the target time τ. Here, [256, 16] and [4, 64] represent the structure of the linear layer, τ represents the time it takes for a pedestrian to change their current walking speed to the desired speed (from the starting position to the target position when there are no obstacles), and [4, 64, 256, 16] and [32, 512, 256, 512, 1] represent the structure of the multilayer perceptron. Further, the attractive force of the target position is calculated based on τ, and the relationship between τ and the attractive force of the target position can be expressed as... Where, p t It is the position of the pedestrian at time t.

[0085] Figure 5 For the second neural network corresponding to the repulsive force between pedestrians, the second neural network encodes the speed and position of other pedestrians in the neighborhood of the target pedestrian, and then inputs them into a long short-term memory network to capture temporal features. After transformation by a linear layer, the output is a function F. collision The first result at time t Furthermore, within the target pedestrian's neighborhood, the velocities and positions of other pedestrians are processed by a multilayer perceptron, outputting the second result of the collision network for the target pedestrian j at time t. Furthermore, the XOR operation result of the first result and the second result is passed through a multilayer perceptron to output the target collision coefficient k. nj Furthermore, based on k nj The repulsive force between the pedestrians, k, was calculated. nj The relationship between the repulsive force between pedestrians can be expressed as: in, in It is the position of pedestrian n at time t. The formula represents the distance r between pedestrian n and pedestrian j at time t, where r is the position of pedestrian j at time t. collision This represents hyperparameters. It's understandable that τ and k...nj They are F goal and F collision Explainable key parameters.

[0086] Furthermore, the third neural network, which interacts with the repulsive forces of the environment, directly learns the target environment parameter k through backpropagation and stochastic gradient descent. env and based on Calculate the repulsive force of the environment; in this formula, p environment It is the location of the obstacle. It is the position of pedestrian n at time t.

[0087] After obtaining the attraction of the target location, the repulsion between pedestrians, and the repulsion of the environment, the target trajectory of the target pedestrian reaching the target location is predicted using the attraction of the target location, the repulsion between pedestrians, and the repulsion of the environment.

[0088] As can be seen, this application proposes a pedestrian trajectory prediction method, including: acquiring pedestrian data, vehicle data, and environmental data collected by a camera, and removing irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal; calculating the target position of the target pedestrian based on the removed pedestrian data, the vehicle data, and the environmental data according to a target classification algorithm; and calculating the target position using the target position attraction, inter-pedestrian repulsion, and environmental repulsion determined based on the target model output to predict the target trajectory of the target pedestrian to the target position. In summary, this application introduces a preset data processing mechanism to remove irregular trajectory data from the pedestrian data, and then predicts the target position of the pedestrian based on the removed pedestrian data, thereby improving the prediction accuracy of the target position. On the other hand, the target model in this application transforms various influencing factors in a complex environment into the expression of "forces," using forces with a certain magnitude and direction to describe the constraints on the target pedestrian, thereby calculating the target trajectory of the target pedestrian in that environment. It is understood that the aforementioned various influencing factors include other pedestrians or obstacles, etc. In this way, this application reduces the possibility of the target pedestrian intersecting with other pedestrians or obstacles in the environment, making the predicted target trajectory more realistic and accurate.

[0089] This application discloses a vehicle control method, which, compared to the previous embodiment, further explains and optimizes the technical solution. See also... Figure 6 As shown, it specifically includes:

[0090] Step S21: Acquire pedestrian data, vehicle data and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal.

[0091] Step S22: Calculate the removed pedestrian data, vehicle data, and environmental data according to the target classification algorithm to predict the target location of the target pedestrian.

[0092] Step S23: Calculate the target location using the target location attraction, inter-pedestrian repulsion, and environmental repulsion determined by the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target location.

[0093] Step S24: Control the vehicle based on the pedestrian data, the vehicle data, the environmental data, and the target trajectory.

[0094] Figure 7 This is a flowchart of a specific vehicle control method disclosed in this application. Figure 8 This is a schematic diagram illustrating a scenario of controlling a vehicle based on a vehicle control method disclosed in this application. Figure 7 The system includes an intelligent forward-looking camera, a data processing module, a target classification algorithm module, a data stitching module, a physical neural network model prediction module, and a vehicle control module. The target classification algorithm module outputs the target location. The data stitching module stitches the outputs from the data processing module and the target classification algorithm, and sends the stitched result to the physical neural network model prediction module, which then outputs the pedestrian trajectory. The vehicle control module receives the pedestrian trajectory output from the physical neural network module and data collected by the intelligent forward-looking camera. Based on the predicted pedestrian trajectory and the data collected by the intelligent camera, it determines whether the pedestrian intersects with the vehicle's trajectory and controls the vehicle's acceleration and deceleration accordingly.

[0095] For example, Figure 8 (a) is a scenario where a pedestrian is about to enter the vehicle's driving trajectory. The intelligent forward-looking camera detects data such as pedestrians, and the physical neural module predicts that the pedestrian will appear in front of the vehicle and there is a risk of collision. Therefore, the vehicle control module decelerates the vehicle in advance to improve vehicle comfort. Figure 8 (b) is a scenario where a pedestrian is entering the vehicle's driving trajectory but is about to leave the vehicle's driving trajectory. After obtaining the output of the physical neural module, the vehicle control module will appropriately give a small deceleration or no deceleration according to the pedestrian's trajectory to improve vehicle comfort.

[0096] In summary, this application improves the safety and comfort of L2+ vehicles passing through traffic light intersections by predicting pedestrian trajectories using a purely visual approach; it improves the accuracy of pedestrian trajectory prediction by classifying whether pedestrians are crossing the intersection; it incorporates a kinematic model into the neural network to explain the changing patterns of the predicted data; it adds pedestrian and environmental repulsion forces to the neural network to reduce the possibility of pedestrian-to-pedestrian data and pedestrian-to-obstacle intersections; and it adds a special trajectory processing mechanism to the data processing module to further improve the model's prediction accuracy.

[0097] Accordingly, this application also discloses a pedestrian trajectory prediction device, see [link to relevant documentation]. Figure 9 As shown, the device includes:

[0098] The data removal module 11 is used to acquire pedestrian data, vehicle data and environmental data collected by the camera, and remove irregular trajectory data in the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal.

[0099] The target location determination module 12 is used to calculate the removed pedestrian data, vehicle data and environmental data according to the target classification algorithm to predict the target location of the target pedestrian;

[0100] The target trajectory determination module 13 is used to calculate the target position using the target position attraction, inter-pedestrian repulsion and environmental repulsion determined by the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target position.

[0101] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0102] As can be seen, this application proposes a pedestrian trajectory prediction device, comprising: a data removal module 11, used to acquire pedestrian data, vehicle data, and environmental data collected by a camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain removed pedestrian data; a target location determination module 12, used to calculate the removed pedestrian data, vehicle data, and environmental data according to a target classification algorithm to predict the target location of the target pedestrian; and a target trajectory determination module 13, used to calculate the target location using the target location attraction, inter-pedestrian repulsion, and environmental repulsion determined based on the results of the target model output, to predict the target trajectory of the target pedestrian to the target location. In summary, this application introduces a preset data processing mechanism and removes irregular trajectory data from the pedestrian data based on this mechanism, and then predicts the target location of the pedestrian based on the removed pedestrian data, thereby improving the prediction accuracy of the target location. On the other hand, the target model in this application transforms various influencing factors in a complex environment into the expression of "forces," using forces with a certain magnitude and direction to describe the constraints on the target pedestrian, thereby calculating the target trajectory of the target pedestrian in that environment. It is understood that the aforementioned various influencing factors include other pedestrians or obstacles, etc. In this way, this application reduces the possibility of the target pedestrian intersecting with other pedestrians or obstacles in the environment, making the predicted target trajectory more realistic and accurate.

[0103] Accordingly, this application also discloses a vehicle control device, see [link to relevant documentation]. Figure 10 As shown, the device includes:

[0104] The data removal module 11 is used to acquire pedestrian data, vehicle data and environmental data collected by the camera, and remove irregular trajectory data in the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal.

[0105] The target location determination module 12 is used to calculate the removed pedestrian data, vehicle data and environmental data according to the target classification algorithm to predict the target location of the target pedestrian;

[0106] The target trajectory determination module 13 is used to calculate the target position using the target position attraction, inter-pedestrian repulsion and environmental repulsion determined based on the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target position;

[0107] The vehicle control module 14 is used to control the vehicle based on the pedestrian data, the vehicle data, the environmental data, and the target trajectory.

[0108] For more detailed information on the working process of each of the above modules, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0109] Furthermore, embodiments of this application also provide an electronic device. Figure 11 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0110] Figure 11 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a display screen 23, an input / output interface 24, a communication interface 25, a power supply 26, and a communication bus 27. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the pedestrian trajectory prediction method and vehicle control method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0111] In this embodiment, the power supply 26 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 25 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 24 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0112] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon can include computer programs 221, and the storage method can be temporary storage or permanent storage. The computer programs 221, in addition to including computer programs capable of performing the pedestrian trajectory prediction method and vehicle control method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks.

[0113] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned pedestrian trajectory prediction method and vehicle control method.

[0114] For the specific steps of this method, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0115] The various embodiments in this application are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. For the same or similar parts between the various embodiments, refer to each other. As for the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and relevant parts can be referred to in the method section.

[0116] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0117] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0118] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0119] The present application provides a detailed description of a pedestrian trajectory prediction method, vehicle control method, device, equipment, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present application. Therefore, the content of this specification should not be construed as a limitation of the present application.

Claims

1. A method for predicting pedestrian trajectories, characterized in that, include: Acquire pedestrian data, vehicle data, and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal; The target location of the target pedestrian is predicted by calculating the removed pedestrian data, vehicle data, and environmental data based on the target classification algorithm. The target location is calculated using the attraction, repulsion between pedestrians, and environmental repulsion determined by the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target location; The step of calculating the target location of the target pedestrian based on the removed pedestrian data, vehicle data, and environmental data using a target classification algorithm includes: Determine the feature vectors corresponding to the removed pedestrian data, vehicle data, and environmental data, and combine the feature vectors to obtain several new feature vectors; The new feature vectors are input into the corresponding base predictors to obtain several prediction probabilities. The total probability of the target is determined based on the plurality of predicted probabilities and the weight value of each predicted probability, and the target position is determined based on the total probability of the target.

2. The pedestrian trajectory prediction method according to claim 1, characterized in that, The method of removing irregular trajectory data from pedestrian data based on a preset data processing mechanism includes: The predicted trajectory data of the target pedestrian in the previous frame and the actual trajectory data in the current frame are determined from the pedestrian data. Determine a first difference between the actual trajectory data of the current frame and the predicted trajectory data of the previous frame, and determine a first target value corresponding to the first difference; If the absolute value of the first target value is greater than the first preset threshold, then the trajectory data of the target pedestrian before the actual trajectory data of the current frame will be removed. And / or, the step of removing irregular pedestrian trajectory data from the pedestrian data based on a preset data processing mechanism includes: The actual trajectory data of the target pedestrian in the previous frame and the actual trajectory data in the current frame are determined from the pedestrian data. Determine a second difference between the actual trajectory data of the current frame and the actual trajectory data of the previous frame, and determine a second target value corresponding to the second difference; If the absolute value of the second target value is greater than the second preset threshold, then the trajectory data of the target pedestrian before the actual trajectory data of the current frame will be removed.

3. The pedestrian trajectory prediction method according to claim 2, characterized in that, Determining the first target value corresponding to the first difference includes: The product of the first difference and the first target scaling ratio is determined as the first target value corresponding to the first difference; Accordingly, determining the second target value corresponding to the second difference includes: The product of the second difference and the second target scaling ratio is determined as the second target value corresponding to the second difference.

4. The pedestrian trajectory prediction method according to claim 1, characterized in that, Before calculating the target location using the target location attraction, inter-pedestrian repulsion, and environmental repulsion determined by the results based on the target model output, the method further includes: A first neural network corresponding to the attractive force at the target location, a second neural network corresponding to the repulsive force between pedestrians, and a third neural network corresponding to the repulsive force of the environment are determined. The target model is constructed based on the first neural network, the second neural network, and the third neural network.

5. The pedestrian trajectory prediction method according to claim 4, characterized in that, The step of calculating the target location using the target location attraction, inter-pedestrian repulsion, and environmental repulsion determined based on the target model output, to predict the target trajectory of the pedestrian reaching the target location, includes: Based on the target time output by the first neural network in the target model, the attractiveness of the target location is calculated; wherein, the target time is the time it takes for the target pedestrian to reach the target location from the current location when there are no obstacles. The repulsive force between pedestrians is calculated based on the target collision coefficient output by the second neural network in the target model. The environmental repulsion force is calculated based on the target environment coefficient output by the third neural network in the target model; The target location is calculated using the attraction force of the target location, the repulsion force between pedestrians, and the repulsion force of the environment to predict the target trajectory of the target pedestrian to reach the target location.

6. A vehicle control method, characterized in that, include: Acquire pedestrian data, vehicle data, and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal; The target location of the target pedestrian is predicted by calculating the removed pedestrian data, vehicle data, and environmental data based on the target classification algorithm. The target location is calculated using the attraction, repulsion between pedestrians, and environmental repulsion determined by the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target location; The vehicle is controlled based on the pedestrian data, the vehicle data, the environmental data, and the target trajectory. The step of calculating the target location of the target pedestrian based on the removed pedestrian data, vehicle data, and environmental data using a target classification algorithm includes: Determine the feature vectors corresponding to the removed pedestrian data, vehicle data, and environmental data, and combine the feature vectors to obtain several new feature vectors; The new feature vectors are input into the corresponding base predictors to obtain several prediction probabilities. The total probability of the target is determined based on the plurality of predicted probabilities and the weight value of each predicted probability, and the target position is determined based on the total probability of the target.

7. A pedestrian trajectory prediction device, characterized in that, include: The data removal module is used to acquire pedestrian data, vehicle data and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal. The target location determination module is used to calculate the removed pedestrian data, vehicle data and environmental data according to the target classification algorithm to predict the target location of the target pedestrian; The target trajectory determination module is used to calculate the target position using the target position attraction, inter-pedestrian repulsion and environmental repulsion determined based on the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target position; The step of calculating the target location of the target pedestrian based on the removed pedestrian data, vehicle data, and environmental data using a target classification algorithm includes: Determine the feature vectors corresponding to the removed pedestrian data, vehicle data, and environmental data, and combine the feature vectors to obtain several new feature vectors; The new feature vectors are input into the corresponding base predictors to obtain several prediction probabilities. The total probability of the target is determined based on the plurality of predicted probabilities and the weight value of each predicted probability, and the target position is determined based on the total probability of the target.

8. A vehicle control device, characterized in that, include: The data removal module is used to acquire pedestrian data, vehicle data and environmental data collected by the camera, and remove irregular trajectory data from the pedestrian data based on a preset data processing mechanism to obtain pedestrian data after removal. The target location determination module is used to calculate the removed pedestrian data, vehicle data and environmental data according to the target classification algorithm to predict the target location of the target pedestrian; The target trajectory determination module is used to calculate the target position using the target position attraction, inter-pedestrian repulsion and environmental repulsion determined based on the results of the target model output, so as to predict the target trajectory of the target pedestrian to the target position; The vehicle control module is used to control the vehicle based on the pedestrian data, the vehicle data, the environmental data, and the target trajectory; The step of calculating the target location of the target pedestrian based on the removed pedestrian data, vehicle data, and environmental data using a target classification algorithm includes: Determine the feature vectors corresponding to the removed pedestrian data, vehicle data, and environmental data, and combine the feature vectors to obtain several new feature vectors; The new feature vectors are input into the corresponding base predictors to obtain several prediction probabilities. The total probability of the target is determined based on the plurality of predicted probabilities and the weight value of each predicted probability, and the target position is determined based on the total probability of the target.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images