Pedestrian trajectory prediction method, device and equipment

Abstract

The present invention provides a method, device and equipment for predicting pedestrian trajectories, the method comprising: obtaining the trajectories of pedestrians in a first time period in an environment; dividing the first time period and the pedestrian trajectories to form a plurality of first time steps and trajectory segments corresponding to each of the first time steps; constructing a trajectory prediction model formed by at least an encoder, a stepwise target estimation and aggregation module, a decoder and a guided sampling module; training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data, so that the trajectory prediction model can predict the probability of pedestrian trajectories corresponding to each second time step in the future second time period; and predicting the trajectories of pedestrians in a specified area based on the trajectory prediction model. The pedestrian trajectory prediction method of the present invention can quickly and accurately predict the trajectories of pedestrians in different scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and specifically relates to a pedestrian trajectory prediction method, device and equipment. Background Art

[0002] Pedestrian trajectory prediction is an important issue in the field of intelligent robots and intelligent transportation. Pedestrian trajectory prediction has important applications in many scenarios. For example, the autonomous driving system should make reasonable predictions about the movements of pedestrians around it like a human driver to prevent aggressive or overly conservative driving. In addition, mobile robots such as self-service robots are often seen in shopping malls or hotels, and their moving routes will not conflict with pedestrians.

[0003] The task of pedestrian trajectory prediction is to predict the future trajectory of pedestrians based on historical trajectories and environmental information. The human brain can often accurately and intuitively predict the short-term trajectory of the future from a historical trajectory of the intelligent body. Whether it is the driver in the car or the dazed person lying on the windowsill watching the crowd, the experience of the brain makes the movement patterns of most external moving bodies understandable and predictable, but it is challenging to let the machine complete this task. On the one hand, if a method based on a knowledge base is designed, although it is highly interpretable, it is a huge challenge to enable the machine to build an accurate movement pattern, because overcoming this challenge requires reliable experience and modeling capabilities, which involves the integration of knowledge such as kinematics, dynamics, and psychology. On the other hand, the future short-term behavior of each pedestrian is uncertain. In recent years, methods based on deep learning have shown great potential. Thanks to the development of RNN and attention mechanisms, the combination of this technology can enable the model to show higher prediction accuracy to a certain extent.

[0004] Specifically, the current multi-modal trajectory prediction (Multimodel Trajectory Prediction) is one of the main challenges of trajectory prediction. In reality, a pedestrian's historical trajectory can correspond to multiple future possibilities, such as turning in different directions or going straight. Multi-modal trajectory prediction is to predict multiple reasonable trajectories. In order to generate multi-modal trajectories, some methods use generative models such as GAN or CVAE, but this method will cause the generated model to have mode collapse problems in some scenarios. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a pedestrian trajectory prediction method, device and equipment for quickly and accurately predicting the trajectory of pedestrians in different scenarios.

[0006] The present invention includes a method for predicting pedestrian trajectories, comprising:

[0007] Obtain the pedestrian trajectory in the first time period in the environment;

[0008] Dividing the first time period and the pedestrian trajectory to form a plurality of first time steps and a trajectory segment corresponding to each of the first time steps;

[0009] Constructing a trajectory prediction model formed by at least an encoder, a stepwise target estimation and aggregation module, a decoder, and a guided sampling module;

[0010] Training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data, so that the trajectory prediction model can predict the probability of a pedestrian trajectory corresponding to each second time step in a future second time period;

[0011] The trajectory of pedestrians in the specified area is predicted based on the trajectory prediction model.

[0012] As an optional embodiment, obtaining the pedestrian trajectory in the first time period in the environment includes:

[0013] Periodically scanning environmental information of the environment, wherein the environmental information includes pedestrian information;

[0014] Processing the environmental information obtained from each scan based on a three-dimensional modeling algorithm to determine coordinate data of pedestrians in the environment in each scan result;

[0015] The pedestrian trajectory is obtained based on the coordinate data of the pedestrian in multiple scanning results.

[0016] As an optional embodiment, the encoder is formed by a gated recursive unit;

[0017] The step of training the trajectory prediction model based on the plurality of first time steps and corresponding trajectory segments as training data includes:

[0018] Encoding the trajectory segment of the first time step based on the encoder, and extracting the temporal features of the encoding result;

[0019] A hidden layer state of the encoder is determined based at least on the encoding result and the time feature.

[0020] As an optional embodiment, the step-by-step target estimation and aggregation module includes a target estimator and a target aggregator;

[0021] The step of training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes:

[0022] Inputting the hidden layer state of the encoder into the target estimator to predict the trajectory target for each second time step based on the target estimator;

[0023] Based on the target aggregator, a weighted sum is performed on the predicted multiple trajectory targets in the time dimension to obtain an aggregated trajectory target corresponding to the first time step.

[0024] As an optional embodiment, the training of the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes:

[0025] The aggregated trajectory target is input into the encoder, so that the encoder updates its hidden layer state based on the encoding result, time features and the aggregated trajectory target corresponding to the next first time step, and the updated hidden layer state is input into the target estimator, so that the target estimator and target aggregator calculate and generate the aggregated trajectory target corresponding to the next first time step based on the hidden layer state updated by the encoder.

[0026] As an optional embodiment, the decoder is formed by a gated recursive unit;

[0027] The step of training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes:

[0028] Initialize the decoder based on the current hidden layer state of the encoder;

[0029] Inputting the aggregate trajectory target corresponding to the current hidden layer state of the encoder to the initialized decoder so that the decoder updates its hidden layer state;

[0030] The updated hidden layer state of the decoder and the aggregated trajectory target are input into the decoder so that the decoder updates its hidden layer state again, and the number of hidden layer state updates of the decoder after each initialization is consistent with the number of second time steps.

[0031] As an optional embodiment, the prediction model further includes a multi-layer perceptron;

[0032] The step of training the trajectory prediction model based on the plurality of first time steps and corresponding trajectory segments as training data includes:

[0033] The updated hidden layer state of the decoder outputted by the multi-layer perceptron is converted into a probability distribution of trajectory points;

[0034] The probability distribution of the trajectory points is input into the guided sampling module to obtain the trajectory prediction value corresponding to each second time step.

[0035] As an optional embodiment, the guided sampling factor of the guided sampling module is the hidden layer state of the encoder and the predicted trajectory target. Corresponding to different first time steps, the hidden layer state of the encoder is different, the predicted trajectory target is different, and the probability distribution of the trajectory point is different.

[0036] Another embodiment of the present invention also provides a pedestrian trajectory prediction device, including:

[0037] An acquisition module, used for acquiring pedestrian trajectories in a first time period in an environment;

[0038] A division module, used for dividing the first time period and the pedestrian trajectory to form a plurality of first time steps and a trajectory segment corresponding to each of the first time steps;

[0039] A construction module, used to construct a trajectory prediction model formed by at least an encoder, a step-by-step target estimation and aggregation module, a decoder, and a guided sampling module;

[0040] A training module, used for training the trajectory prediction model according to the plurality of the first time steps and the corresponding trajectory segments as training data, so that the trajectory prediction model can predict the probability of the pedestrian trajectory corresponding to each second time step in the future second time period;

[0041] The trajectory of pedestrians in the specified area is predicted based on the trajectory prediction model.

[0042] Another embodiment of the present invention further provides an electronic device, including:

[0043] at least one processor; and,

[0044] a memory communicatively coupled to the at least one processor;

[0045] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the pedestrian trajectory prediction method as described in any one of the embodiments above.

[0046] The beneficial effect of the present invention is that by constructing a trajectory prediction model formed by at least an encoder, a step-by-step target estimation and aggregation module, a decoder, and a guided sampling module, and training the trajectory prediction model through training data formed by dividing the collected pedestrian trajectories into time steps, the trajectory prediction model can be used alone to perform efficient and accurate multimodal prediction of the motion trajectories of pedestrians in different scenarios, and the application range is wide and easy to implement, including configuring the trajectory prediction model in vehicles and service robots to realize automatic driving of vehicles and autonomous movement of service robots.

[0047] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings.

[0048] The technical solution of the present application is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 It is a schematic flow chart of the pedestrian trajectory prediction method of the present invention.

[0051] Figure 2 The figure is a schematic application flow chart of the pedestrian trajectory prediction method of the present invention.

[0052] Figure 3 FIG. 4 is another schematic application flow chart of the pedestrian trajectory prediction method of the present invention.

[0053] Figure 4 It is a schematic structural block diagram of the pedestrian trajectory prediction device of the present invention. DETAILED DESCRIPTION

[0054] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings, but are not intended to limit the present invention.

[0055] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope and spirit of the present disclosure will occur to those skilled in the art.

[0056] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the present application and, together with the general description of the present disclosure given above and the detailed description of the embodiments given below, serve to explain the principles of the present invention.

[0057] These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the accompanying drawings.

[0058] It should also be understood that, although the present application has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the present invention that have the characteristics described in the claims and are therefore within the scope of protection defined thereby.

[0059] The above and other aspects, features and advantages of the present invention will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0060] Specific embodiments of the present invention are described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present invention, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid obscuring the present disclosure with unnecessary or redundant details. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present disclosure in a variety of ways with substantially any suitable detailed structure.

[0061] This specification may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," all of which may refer to one or more of the same or different embodiments according to the present disclosure.

[0062] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0063] like Figure 1 As shown, an embodiment of the present invention provides a pedestrian trajectory prediction method, comprising:

[0064] S1: Obtain the pedestrian trajectory in the first time period of the environment;

[0065] S2: dividing the first time period and the pedestrian trajectory to form a plurality of first time steps and a trajectory segment corresponding to each first time step;

[0066] S3: construct a trajectory prediction model formed by at least an encoder, a stepwise target estimation and aggregation module, a decoder, and a guided sampling module;

[0067] S4: training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data, so that the trajectory prediction model can predict the probability of the pedestrian trajectory corresponding to each second time step in the future second time period;

[0068] S5: Predicting the trajectory of pedestrians in the designated area based on the trajectory prediction model.

[0069] Among them, the first time step and the second time step can be the same time or different time, such as predicting the trajectory data of the long second time step based on the trajectory data of the short first time step, which is not unique. Based on the above content, it can be known that this embodiment constructs a trajectory prediction model formed by at least an encoder, a step-by-step target estimation and aggregation module, a decoder, and a guided sampling module, and trains the trajectory prediction model through the training data formed by dividing the collected pedestrian trajectory by time steps. It can only use the trajectory prediction model to perform efficient and accurate multi-modal prediction of the motion trajectory of pedestrians in different scenarios, and has a wide range of applications and is easy to implement. For example, the trajectory prediction model is configured in a vehicle or a service robot to realize the automatic driving of the vehicle and the autonomous movement of the service robot. The pedestrian trajectory prediction method in this embodiment is mainly used in the automatic driving of vehicles, such as the automatic driving of small-sized vehicles for transportation, to predict the movement trajectory of pedestrians, and then to dodge and avoid pedestrians in time through prediction to prevent collisions.

[0070] Furthermore, the encoder and decoder in this embodiment are both formed by a gated recurrent unit (GRU), and by introducing a guided sampling module, the trajectory prediction model completely avoids using a conditional variational encoder. Therefore, the trajectory prediction model in this embodiment does not have a mode collapse problem, and the introduction of the guided sampling module can achieve high-quality sampling of the prediction results in the prediction stage, so that all important features related to the prediction can be taken into account without omissions, effectively ensuring the training accuracy and prediction accuracy of the model.

[0071] In this embodiment, obtaining the pedestrian trajectory in the first time period in the environment includes:

[0072] S6: Periodically scan environmental information of the environment, where the environmental information includes pedestrian information;

[0073] S7: Processing the environment information obtained from each scan based on a three-dimensional modeling algorithm to determine coordinate data of pedestrians in the environment in each scan result;

[0074] S8: Obtaining the pedestrian trajectory based on the coordinate data of the pedestrian in multiple scanning results.

[0075] Taking a vehicle as an example, when applied, the vehicle's sensors, such as a laser radar, etc., can be used to obtain information about the vehicle's surrounding environment, and the preset three-dimensional modeling algorithm in the vehicle system can be used for calculation to obtain the coordinates of pedestrians around the vehicle, and then the coordinate data of each pedestrian is scanned and updated periodically or in real time to generate pedestrian trajectories of the surrounding pedestrians. The pedestrian trajectory can be used to train the model and the subsequent calibration model. When the model is used to predict pedestrian trajectories, the pedestrian trajectory is also obtained as model input data based on the above method to obtain the prediction result. The vehicle plans the path in advance based on the prediction result, and controls the vehicle to travel based on the planned path to ensure traffic safety. That is, the method in this embodiment can support the vehicle's online real-time training model and the subsequent calibration model, that is, to achieve the effect of training while driving, so that the user does not need to collect data for training separately, which simplifies the training process and achieves zero user operation.

[0076] Furthermore, the step of training the trajectory prediction model based on the plurality of first time steps and corresponding trajectory segments as training data includes:

[0077] S9: encoding the trajectory segment of the first time step based on the encoder, and extracting the time feature of the encoding result;

[0078] S10: Determine a hidden layer state of the encoder based at least on the encoding result and the time feature.

[0079] like Figure 2 As shown, the backbone network in this embodiment is composed of an encoder and a decoder. The encoder is used to encode the collected pedestrian trajectories time by time. Assume that the length of the first time period, that is, the length of the historical time period, is T obs , the length of the second time period, that is, the future time period, is T pred ,The hidden state of the encoder estimates the future T through step-by-step target estimation and aggregation module pred 's goal, that is, to estimate the goal step by step.

[0080] Specifically, the step-by-step target estimation and aggregation module in this embodiment includes a target estimator and a target aggregator;

[0081] The step of training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes:

[0082] S11: inputting the hidden layer state of the encoder into the target estimator to predict the trajectory target for each second time step based on the target estimator;

[0083] S12: performing weighted summation on the predicted multiple trajectory targets in the time dimension based on the target aggregator to obtain an aggregated trajectory target corresponding to the first time step.

[0084] In this embodiment, the target estimator predicts the trajectory target step by step based on GRU and is connected to the fully connected layer, so that the predicted trajectory target at each moment is processed based on the fully connected layer, and then the target aggregator uses the attention mechanism to assign different weights to the predicted trajectory targets at different moments, and finally uses weighted summation to obtain the aggregated trajectory target after compressing the time dimension. The number of aggregated trajectory targets is consistent with the number of first time steps. Each time the encoder processes a trajectory segment corresponding to the first time step, it will output a hidden layer state. The target estimator performs target estimation based on the hidden layer state, and then the target aggregator aggregates to form an aggregated trajectory target.

[0085] Furthermore, the training of the trajectory prediction model based on the plurality of the first time steps and the corresponding trajectory segments as training data further includes:

[0086] S13: Input the aggregated trajectory target into the encoder, so that the encoder updates its hidden layer state based on the encoding result, time features and the aggregated trajectory target corresponding to the next first time step, and inputs the updated hidden layer state into the target estimator, so that the target estimator and target aggregator calculate and generate the aggregated trajectory target corresponding to the next first time step based on the hidden layer state updated by the encoder.

[0087] That is, the aggregated trajectory target of each first time step will be fed back to the encoder and participate in the encoder's processing of the trajectory segment corresponding to the next first time step, so that the encoder can learn in combination with the aggregated trajectory target of the previous first time step and update the hidden layer state. Such iterative calculation can make the encoder's hidden layer state contain as many relevant features of the pedestrian trajectory as possible, so that subsequent target estimators can perform more accurate trajectory target estimation based on the hidden layer state containing more features, laying the foundation for improving the model prediction accuracy.

[0088] As mentioned above, the decoder in this embodiment is formed by a gated recursive unit, and the decoder is mainly used to iteratively generate future trajectories. Specifically, the trajectory prediction model is trained based on the plurality of first time steps and the corresponding trajectory segments as training data, and further includes:

[0089] S14: Initialize the decoder based on the current hidden layer state of the encoder;

[0090] S15: inputting the aggregated trajectory target corresponding to the current hidden layer state of the encoder to the initialized decoder, so that the decoder updates its hidden layer state;

[0091] S16: Inputting the updated hidden layer state of the decoder and the aggregated trajectory target into the decoder, so that the decoder updates its hidden layer state again, and the number of hidden layer state updates after each initialization of the decoder is consistent with the number of second time steps.

[0092] Furthermore, the trajectory prediction model in this embodiment also includes a multi-layer perceptron for assisting the model in generating multimodal trajectories. Accordingly, the trajectory prediction model is trained based on the plurality of first time steps and corresponding trajectory segments as training data, including:

[0093] S17: converting the updated hidden layer state output by the decoder based on the multi-layer perceptron into a probability distribution of trajectory points;

[0094] S18: Input the probability distribution of the trajectory points into the guided sampling module to obtain the trajectory prediction value corresponding to each second time step.

[0095] The probability distribution of the trajectory points in this embodiment is a Gaussian distribution. Based on the above content, it can be known that the decoder is a prediction head that outputs a Gaussian distribution, which is used to predict the Gaussian probability distribution of pedestrian trajectory points at each second time step, and then introduced into the guided sampling module for more reliable sampling, and finally obtains the pedestrian trajectory prediction result, that is, the trajectory prediction value. Specifically, this example converts the hidden layer state of the decoder into the probability distribution of trajectory points. The first input of the decoder is the current hidden layer state of the encoder and the current aggregated trajectory target, that is, the hidden layer state output to the target estimator and the aggregated trajectory target formed based on the hidden layer state. The subsequent encoder inputs are the corresponding hidden layer state when it predicts the probability distribution of the trajectory points at the previous moment and the above-mentioned aggregated trajectory target, that is, the aggregated trajectory target remains unchanged, and only the hidden layer state of the decoder changes in each iteration. Among them, the hidden layer state of the decoder is linearly transformed before being input into the decoder again.

[0096] Based on the above content, the workflow of the trajectory prediction model in this embodiment can be briefly described as follows:

[0097] Input: Pedestrian trajectory X.

[0098] Output: Probability distribution of trajectory for each future time step (second time step)

[0099] X is embedded in high-dimensional features and then input to the encoder, which iteratively extracts the temporal features of X. The target estimator uses the hidden layer state of the encoder to predict the future Tpred The trajectory target of the second time step is obtained; based on the target aggregator, the trajectory targets of multiple time steps are aggregated in the time dimension to obtain the aggregated trajectory target; the decoder is initialized with the encoder hidden layer state; the aggregated trajectory target is input to the decoder to update the decoder hidden layer state. Among them, the input of the initialized decoder is the previously calculated aggregated trajectory target and the high-dimensional embedded data of the decoder hidden layer state at the previous time step; the hidden layer state of the decoder is output through the multi-layer perceptron (MLP) to predict the trajectory path point probability distribution T pred ; Update the hidden layer state of the encoder, including the input aggregate trajectory target and the high-dimensional embedding feature of X at the current time step as input, to achieve the update of the hidden layer state of the encoder; when the encoder iterates and processes the last trajectory segment corresponding to the first time step in sequence, the trajectory probability distribution output by the decoder is the prediction result, which includes the probability of all trajectory prediction points at the second time step.

[0100] Furthermore, in this embodiment, if Figure 3 As shown, the guided sampling factor of the guided sampling module (NPSN) is the hidden layer state of the encoder and the predicted trajectory target. Corresponding to different first time steps, the hidden layer state of the encoder is different, the predicted trajectory target is different, and the probability distribution of the trajectory point is different.

[0101] Specifically, the decoder corresponds to the hidden layer state of each second time step, and outputs a five-dimensional prediction result after linear transformation, which indicates a two-dimensional Gaussian distribution, representing the distribution probability of the trajectory point at the current moment:

[0102]

[0103] in, is the distribution probability of the moment prediction, t is the time step of the prediction stage, is the decoder hidden state. f(·) is the MLP layer, which converts the hidden state into the prediction result. Based on the binary Gaussian distribution, the factor guiding the sampling adopts the hidden state of the last observation frame obtained in the encoder stage. Depend on Rough target estimate Predictions for every moment μ, σ, and ρ represent the mean, standard deviation, and correlation coefficient in the X and Y directions of the binary Gaussian distribution. The structure of NPSN includes three multi-layer perceptrons, which are used to respectively Embedding high-dimensional features and converting the fused features of the two into sampled latent vectors, the workflow of the NPSN module is expressed as:

[0104]

[0105]

[0106] V sample =σ v (MLP(cat(F h ,F g )))

[0107] Here, MLP stands for multi-layer perceptron process. The symbols flatten(·) and cat(·) respectively represent the flattening of the target to one dimension (originally with time and space dimensions) and matrix concatenation operations. v is the hyperbolic tangent function. V sample is an N×2 vector, corresponding to Figure 3 The latent vector in V is the potential vector, which determines the position of the sample in the two-dimensional Gaussian distribution. N is the number of samples extracted (the number of modes), and 2 is the XY dimension of the trajectory. sample The matrix dimensions are reorganized to obtain the coefficients of each mode, which are combined with the parameters of the Gaussian distribution to produce trajectory point samples, that is, to produce prediction results.

[0108] Furthermore, the loss function of the model includes the predicted binary Gaussian distribution loss and NPSN loss in this embodiment. The distribution loss aims to make the Gaussian distribution more reasonable, and is defined as the negative logarithmic natural loss:

[0109]

[0110] Where W SGNet are all the parameters to be trained in the trajectory prediction model except NPSN, (x t ,y t ) is the true value of the trajectory point. The NPSN loss includes the error with the closest true value and the distance between samples:

[0111]

[0112]

[0113] L NPSN =λ1L dist +λ2L disc

[0114] Among them, W NPSN is the parameter of the NPSN module, M is the number of pedestrians, and N is the number of samples extracted. ||·|| L2 To find the L2 norm. disc The loss is to penalize the two samples with the smallest distance by the L2 norm. NPSN For Ldist and L disc The linear summation of the two errors, in this embodiment, is performed by setting λ1 = 1.0, λ2 = 0.01 to achieve a balance between the two terms.

[0115] like Figure 4 As shown, another embodiment of the present invention also provides a pedestrian trajectory prediction device 100, including:

[0116] An acquisition module, used for acquiring pedestrian trajectories in a first time period in an environment;

[0117] A division module, used for dividing the first time period and the pedestrian trajectory to form a plurality of first time steps and a trajectory segment corresponding to each of the first time steps;

[0118] A construction module, used to construct a trajectory prediction model formed by at least an encoder, a step-by-step target estimation and aggregation module, a decoder, and a guided sampling module;

[0119] A training module, used for training the trajectory prediction model according to the plurality of the first time steps and the corresponding trajectory segments as training data, so that the trajectory prediction model can predict the probability of the pedestrian trajectory corresponding to each second time step in the future second time period;

[0120] The prediction module is used to predict the trajectory of pedestrians in a specified area according to the trajectory prediction model.

[0121] As an optional embodiment, obtaining the pedestrian trajectory in the first time period in the environment includes:

[0122] Periodically scanning environmental information of the environment, wherein the environmental information includes pedestrian information;

[0123] Processing the environmental information obtained from each scan based on a three-dimensional modeling algorithm to determine coordinate data of pedestrians in the environment in each scan result;

[0124] The pedestrian trajectory is obtained based on the coordinate data of the pedestrian in multiple scanning results.

[0125] As an optional embodiment, the encoder is formed by a gated recursive unit;

[0126] The step of training the trajectory prediction model based on the plurality of first time steps and corresponding trajectory segments as training data includes:

[0127] Encoding the trajectory segment of the first time step based on the encoder, and extracting the temporal features of the encoding result;

[0128] A hidden layer state of the encoder is determined based at least on the encoding result and the time feature.

[0129] As an optional embodiment, the step-by-step target estimation and aggregation module includes a target estimator and a target aggregator;

[0130] The step of training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes:

[0131] Inputting the hidden layer state of the encoder into the target estimator to predict the trajectory target for each second time step based on the target estimator;

[0132] Based on the target aggregator, a weighted sum is performed on the predicted multiple trajectory targets in the time dimension to obtain an aggregated trajectory target corresponding to the first time step.

[0133] As an optional embodiment, the training of the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes:

[0134] The aggregated trajectory target is input into the encoder, so that the encoder updates its hidden layer state based on the encoding result, time features and the aggregated trajectory target corresponding to the next first time step, and the updated hidden layer state is input into the target estimator, so that the target estimator and target aggregator calculate and generate the aggregated trajectory target corresponding to the next first time step based on the hidden layer state updated by the encoder.

[0135] As an optional embodiment, the decoder is formed by a gated recursive unit;

[0136] The step of training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes:

[0137] Initialize the decoder based on the current hidden layer state of the encoder;

[0138] Inputting the aggregate trajectory target corresponding to the current hidden layer state of the encoder to the initialized decoder so that the decoder updates its hidden layer state;

[0139] The updated hidden layer state of the decoder and the aggregated trajectory target are input into the decoder so that the decoder updates its hidden layer state again, and the number of hidden layer state updates of the decoder after each initialization is consistent with the number of second time steps.

[0140] As an optional embodiment, the prediction model further includes a multi-layer perceptron;

[0141] The step of training the trajectory prediction model based on the plurality of first time steps and corresponding trajectory segments as training data includes:

[0142] The updated hidden layer state of the decoder outputted by the multi-layer perceptron is converted into a probability distribution of trajectory points;

[0143] The probability distribution of the trajectory points is input into the guided sampling module to obtain the trajectory prediction value corresponding to each second time step.

[0144] As an optional embodiment, the guided sampling factor of the guided sampling module is the hidden layer state of the encoder and the predicted trajectory target. Corresponding to different first time steps, the hidden layer state of the encoder is different, the predicted trajectory target is different, and the probability distribution of the trajectory point is different.

[0145] Another embodiment of the present invention further provides an electronic device, including:

[0146] at least one processor; and,

[0147] a memory communicatively coupled to the at least one processor;

[0148] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the pedestrian trajectory prediction method as described in any one of the embodiments above.

[0149] Another embodiment of the present invention further provides a storage medium, which includes a stored program, wherein when the program is running, a device including the storage medium is controlled to execute the pedestrian trajectory prediction method as described in any of the embodiments above.

[0150] The embodiment of the present invention also provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions, and when the computer-executable instructions are executed, at least one processor executes a pedestrian trajectory prediction method such as in the embodiment described above. It should be understood that each solution in this embodiment has the corresponding technical effect in the above method embodiment, which will not be repeated here.

[0151] It should be noted that the computer storage medium of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program configured to be used by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, antenna, optical cable, RF, etc., or any suitable combination of the above.

[0152] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0153] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A pedestrian trajectory prediction method, characterized in that: include: Obtain the pedestrian trajectory in the first time period in the environment; Dividing the first time period and the pedestrian trajectory to form a plurality of first time steps and a trajectory segment corresponding to each of the first time steps; Constructing a trajectory prediction model formed by at least an encoder, a stepwise target estimation and aggregation module, a decoder, and a guided sampling module; Training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data, so that the trajectory prediction model can predict the probability of a pedestrian trajectory corresponding to each second time step in a future second time period; Predicting the trajectory of pedestrians in a specified area based on the trajectory prediction model; The encoder is formed by a gated recursive unit and is used to encode the trajectory of the first time step. The step-by-step target estimation and aggregation module includes a target estimator and a target aggregator, which are used to perform trajectory prediction on the encoded trajectory and determine the aggregated trajectory target. The decoder is formed by a gated recursive unit and is used to update its own hidden layer using the aggregated trajectory target. The guided sampling module is used to predict the trajectory prediction value of the second time step using the hidden layer data updated by the encoder.

2. The pedestrian trajectory prediction method according to claim 1, characterized in that: The obtaining of pedestrian trajectories in a first time period in the environment includes: Periodically scanning environmental information of the environment, wherein the environmental information includes pedestrian information; Processing the environmental information obtained from each scan based on a three-dimensional modeling algorithm to determine coordinate data of pedestrians in the environment in each scan result; The pedestrian trajectory is obtained based on the coordinate data of the pedestrian in multiple scanning results.

3. The pedestrian trajectory prediction method according to claim 1, characterized in that: The step of training the trajectory prediction model based on the plurality of first time steps and corresponding trajectory segments as training data includes: Encoding the trajectory segment of the first time step based on the encoder, and extracting the temporal features of the encoding result; A hidden layer state of the encoder is determined based at least on the encoding result and the time feature.

4. The pedestrian trajectory prediction method according to claim 3, characterized in that: The step of training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes: Inputting the hidden layer state of the encoder into the target estimator to predict the trajectory target for each second time step based on the target estimator; Based on the target aggregator, a weighted sum is performed on the predicted multiple trajectory targets in the time dimension to obtain an aggregated trajectory target corresponding to the first time step.

5. The pedestrian trajectory prediction method according to claim 4, characterized in that: The step of training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes: The aggregated trajectory target is input into the encoder, so that the encoder updates its hidden layer state based on the encoding result, time features and the aggregated trajectory target corresponding to the next first time step, and the updated hidden layer state is input into the target estimator, so that the target estimator and target aggregator calculate and generate the aggregated trajectory target corresponding to the next first time step based on the hidden layer state updated by the encoder.

6. The pedestrian trajectory prediction method according to claim 4 or 5, characterized in that: The step of training the trajectory prediction model based on the plurality of first time steps and the corresponding trajectory segments as training data further includes: Initialize the decoder based on the current hidden layer state of the encoder; Inputting the aggregate trajectory target corresponding to the current hidden layer state of the encoder to the initialized decoder so that the decoder updates its hidden layer state; The updated hidden layer state of the decoder and the aggregated trajectory target are input into the decoder so that the decoder updates its hidden layer state again, and the number of hidden layer state updates of the decoder after each initialization is consistent with the number of second time steps.

7. The pedestrian trajectory prediction method according to claim 6, characterized in that: The prediction model also includes a multi-layer perceptron; The step of training the trajectory prediction model based on the plurality of first time steps and corresponding trajectory segments as training data includes: The updated hidden layer state of the decoder outputted by the multi-layer perceptron is converted into a probability distribution of trajectory points; The probability distribution of the trajectory points is input into the guided sampling module to obtain the trajectory prediction value corresponding to each second time step.

8. The pedestrian trajectory prediction method according to claim 7, characterized in that: The guided sampling factor of the guided sampling module is the hidden layer state of the encoder and the predicted trajectory target. Corresponding to different first time steps, the hidden layer state of the encoder is different, the predicted trajectory target is different, and the probability distribution of the trajectory point is different.

9. A pedestrian trajectory prediction device, characterized in that: include: An acquisition module, used for acquiring pedestrian trajectories in a first time period in an environment; A division module, used for dividing the first time period and the pedestrian trajectory to form a plurality of first time steps and a trajectory segment corresponding to each of the first time steps; A construction module, used to construct a trajectory prediction model formed by at least an encoder, a step-by-step target estimation and aggregation module, a decoder, and a guided sampling module; A training module, used for training the trajectory prediction model according to the plurality of the first time steps and the corresponding trajectory segments as training data, so that the trajectory prediction model can predict the probability of the pedestrian trajectory corresponding to each second time step in the future second time period; A prediction module, used to predict the trajectory of pedestrians in a specified area according to the trajectory prediction model; The encoder is formed by a gated recursive unit and is used to encode the trajectory of the first time step. The step-by-step target estimation and aggregation module includes a target estimator and a target aggregator, which are used to perform trajectory prediction on the encoded trajectory and determine the aggregated trajectory target. The decoder is formed by a gated recursive unit and is used to update its own hidden layer using the aggregated trajectory target. The guided sampling module is used to predict the trajectory prediction value of the second time step using the hidden layer data updated by the encoder.

10. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the pedestrian trajectory prediction method as described in any one of claims 1-8.

Images