A pedestrian trajectory prediction method integrating moving object analysis

Abstract

The present invention provides a pedestrian trajectory prediction method integrating moving target analysis, belonging to the field of pedestrian trajectory prediction. The method of the present invention designs a multi-precision pedestrian feature representation method, uses a gated recurrent unit (GRU) to encode the pedestrian's historical position information, constructs a main network-subnetwork joint pedestrian trajectory prediction model, and designs three sub-networks to complete three sub-tasks: predicting the fine-precision coordinates and coarse-precision coordinates of the pedestrian's end point, and scoring each sub-region in the scene; designs the main network to predict the pedestrian's end point coordinates, and uses the attention mechanism to fuse the output vectors of the main network and the sub-networks; finally, uses the conditional variational autoencoder to decode the encoded state to predict the pedestrian's future multi-modal trajectory. The present invention improves the robustness of pedestrian intention analysis, solves the problem of unclear intermediate intentions of pedestrians, and the predicted end point coordinates can gradually approach the true target and are basically completely fitted with the true trajectory.

Description

Technical Field

[0001] The present invention belongs to the field of pedestrian trajectory prediction, and relates to the acquisition of pedestrian motion characteristics, the analysis of pedestrian motion intentions, and the modeling of dynamic scene information. Specifically, it relates to a pedestrian trajectory prediction method that integrates moving target analysis. Background Art

[0002] Trajectory prediction refers to predicting the possible motion trajectories of agents such as target pedestrians, vehicles, and robots in a future period based on their historical motion states. Trajectory prediction algorithms are an important part of unmanned driving and robot navigation technologies and are also one of the current hot research directions. With the commercialization of 5G networks and the development of urban modernization, the country attaches great importance to the construction of key directions such as smart cities, public transportation, and intelligent robots. The Ministry of Industry and Information Technology pointed out in 2021 that it is necessary to strengthen research on core technologies and break through common technologies such as robot information perception and navigation. As a key basic research technology, pedestrian trajectory prediction algorithms play a crucial role in real-world scenarios such as robot navigation and pedestrian intention analysis. In order to make the movement of robots in complex scenarios closer to that of humans and avoid collisions with pedestrians, robots need to efficiently predict the motion intentions of nearby pedestrians based on information such as the position, movement direction, speed, and acceleration of pedestrians, and then perform effective obstacle avoidance operations. Surveillance systems and intelligent tracking modules for urban public safety also need to analyze the movement and behavior of crowds to understand the interaction patterns of crowds, and issue early warnings for prohibited areas that pedestrians may enter or violations that may occur, so as to better manage infrastructure and optimize resource allocation. Therefore, the problem of crowd trajectory prediction has practical significance, is relatively complex, and has relatively high research value.

[0003] Trajectory prediction is a basic research content and an important link in path planning. One of the key requirements is to analyze the intention targets of pedestrians, such as whether surrounding vehicles need to change lanes and whether the pedestrian in front needs to cross the road. The end target of a pedestrian, as an embodiment of the motion intention, is also an important factor affecting the future trajectory. However, predicting the end target of a pedestrian is a very challenging task. Since the scene information where the pedestrian is located is dynamically changing, the model not only needs to consider the pedestrian's own historical motion state, but also needs to dynamically model the possible future motion states of the pedestrian by combining the continuously changing scene information, and then plan the general end intention target of the pedestrian.

[0004] Currently, the process of existing trajectory prediction methods generally can be divided into three stages: (1) encoding the pedestrian's motion state information based on the pedestrian's historical position coordinates; (2) analyzing the pedestrian's intended target position according to the pedestrian's motion state encoded in the first stage; (3) decoding based on the modeling data of the first and second stages to predict the future trajectory. In recent years, many deep learning-based methods have been proposed to improve the above process, and the technologies related to the present invention are mainly the PECNet model and the SGNet model. The two models can refer to the following documents 1 and 2 respectively.

[0005] PECNet (Predicted Endpoint Conditioned Network) is a predicted endpoint conditioned network model that decomposes the trajectory prediction problem into two sub-problems: predicting the pedestrian's intended endpoint and fitting the future complete trajectory. First, PECNet encodes the pedestrian's historical motion state using a multi-layer perceptron (MLP), and then uses an endpoint VAE (variational auto-encoder) module to predict the latent distribution of the pedestrian's endpoint. The model samples from the possible latent distributions of the endpoint to obtain the pedestrian's endpoint position. In the trajectory generation stage, the model directly uses an MLP model to predict the future trajectory by combining three feature information: the surrounding pedestrian interaction information, the pedestrian's historical motion information, and the pedestrian's intended target coordinates predicted in the first stage. However, PECNet has the following disadvantages: This method has a simple assumption that the pedestrian's intended target is determined and will not change with the change of the movement moment. PECNet also only predicts a fixed target and cannot sample the optimal candidate endpoint in the test stage. However, in real life, the pedestrian's motion intention is not a fixed two-dimensional coordinate. On the contrary, the pedestrian's motion intention will change dynamically with the pedestrian's movement. As the movement moment changes, the scene layout information around the pedestrian will also change, and the pedestrian will dynamically pay attention to the possible impact of the change of scene information in different regions on their future movement, and then continuously adjust their endpoint intention target.

[0006] SGNet (Stepwise goal-driven networks) recognizes that a pedestrian's intended goal may change over time during motion. Modeling this goal can provide more accurate and detailed information for future trajectory estimation. Therefore, unlike previous work that focused solely on modeling a single long-term goal, SGNet emphasizes a new prediction direction, proposing a method that incrementally predicts goals to drive trajectory prediction, evaluating and utilizing goals at multiple timescales. SGNet consists of three components: an encoder module that captures historical information, a stepwise goal estimator that predicts future continuous goals, and a decoder module that predicts future trajectories. These goals are effectively integrated into the encoder and decoder modules through an attention mechanism. While SGNet proposes dynamically predicting a pedestrian's intended goal and then fitting a complete trajectory, this approach only utilizes the two-dimensional physical coordinates of the destination goal. This approach, according to the present invention, cannot accurately analyze a pedestrian's motion intention. This is because in real life, a pedestrian's true intention is not a fixed two-dimensional coordinate, but rather a local region within which the two-dimensional coordinate resides. The observable two-dimensional coordinates of the destination cannot fully describe the pedestrian's intention. In particular, existing trajectory prediction research typically uses a model to predict a segment of a pedestrian's complete trajectory. While a pedestrian's complete trajectory may have a clear destination, a segment lacks a clear 2D coordinate of its intended location. Therefore, it's necessary to improve the method of analyzing pedestrian intent using only the 2D destination coordinates. Instead, the model can extract more information features from the local area where the 2D destination coordinates are located to help the model accurately predict the pedestrian's target area.

[0007] Reference 1: Mangalam K, Girase H, Agarwal S, et al. It is not the journey but the destination: Endpoint conditioned trajectory prediction[C] / / EuropeanConference on Computer Vision. Springer, Cham, 2020:759-776.

[0008] Reference 2: Wang C, Wang Y, Xu M, et al. Stepwise goal-driven networks fortrajectory prediction [J]. IEEE Robotics and Automation Letters, 2022. Summary of the Invention

[0009] When the requirement for the prediction accuracy of pedestrian target intention is relatively high, and the existing methods have problems of ignoring the dynamic changes of pedestrian intention and only using two-dimensional spatial coordinates to analyze pedestrian intention with insufficient features, the present invention proposes a pedestrian trajectory prediction method integrating moving target analysis by means of deep learning technology. Through two modules of main network motion pattern modeling and sub-network pedestrian intention dynamic analysis, the intention analysis of pedestrians is made more accurate and reasonable, achieving the effect of improving the prediction accuracy of pedestrian trajectories, and having the advantages of being able to analyze pedestrian intention from multiple dimensions and ensuring accurate prediction of pedestrian intention coordinates.

[0010] A pedestrian trajectory prediction method integrating moving target analysis provided by the present invention includes the following steps:

[0011] Step 1: Extract multi-precision data from the pedestrian historical trajectory video, including the coarse-precision coordinate information, fine-precision coordinate information, and dynamic scene information of the pedestrian;

[0012] Among them, the coarse-precision coordinate is the sub-region coordinate where the pedestrian is located after dividing the scene area, and the fine-precision coordinate refers to the coordinate of the pedestrian in the scene; both the coarse-precision coordinate information and the fine-precision coordinate information include the pedestrian coordinate position, speed, and acceleration.

[0013] Step 2: Construct a main network-subnetwork joint pedestrian trajectory prediction model;

[0014] The sub-network of the pedestrian trajectory prediction model encodes the three types of data of the input pedestrian coarse-precision coordinate information, pedestrian fine-precision coordinate information, and dynamic scene information, and then uses the gated recurrent neural network model GRU e , coarse , score , sub , fine for sequence modeling, predicts and outputs the pedestrian motion intention vector at the current moment, and then inputs the pedestrian motion intention vector into three sub-task models; the first sub-task model uses a multi-layer perceptron f fine to map the pedestrian motion intention vector to obtain the fine-precision coordinate of the pedestrian's end intention; the second sub-task model uses a multi-layer perceptron f coarse to map the pedestrian motion intention vector to obtain the coarse-precision coordinate of the pedestrian's end; the third sub-task model first uses a matrix W score to map the pedestrian motion intention vector, then connects the ReLU activation function, and then uses the Softmax function to score each region in the scene.

[0015] The main network of the pedestrian trajectory prediction model uses a multi-layer perceptron f eAfter encoding the input pedestrian fine-precision coordinate information, it is input into the gated recurrent neural network model GRU together with the target intention vector of the pedestrian at the current moment. The GRU outputs the pedestrian motion state vector of the pedestrian at the current moment, and then uses the multi-layer perceptron f goal Map the output of the GRU to predict the pedestrian's end coordinates.

[0016] The target intention vector of the pedestrian is obtained by fusing and calculating the output of the GRU of the sub-network sub and the output of the GRU of the main network using the attention mechanism.

[0017] Step 3: In the trajectory decoding stage, use the conditional variational autoencoder CVAE to generate multi-modal trajectories.

[0018] The method of the present invention jointly trains the pedestrian trajectory prediction model constructed in step 2 and the CVAE in step 3, and uses the trained model to predict the pedestrian trajectory.

[0019] The advantages and positive effects of the present invention are as follows:

[0020] (1) The method of the present invention proposes a brand-new pedestrian data representation method. Traditional methods usually use two-dimensional spatial coordinates to represent the intention information of pedestrians. However, two-dimensional coordinates cannot fully represent the intention information of pedestrians. Since in most cases, the future trajectory predicted by the model is only a segment of the complete trajectory of the pedestrian, the pedestrian may have a clear end coordinate, but there is no definite intermediate point position. Therefore, the present invention proposes to use the information of three dimensions of pedestrian fine-precision coordinates, coarse-precision coordinates, and adjacent local scenes to represent the position information of pedestrians, which can improve the robustness of pedestrian intention analysis and solve the problem of unclear intermediate intentions of pedestrians.

[0021] (2) The method of the present invention proposes a multi-task-based dynamic analysis sub-network for pedestrian intentions, designs loss functions for three sub-tasks to update the network parameters, and uses the attention mechanism to fuse the end-point prediction information of the sub-tasks with the pedestrian motion information of the backbone network to assist the backbone network in fitting the complete trajectory, achieving the dynamic effect of the predicted intention gradually approaching the real target. The present invention uses a dedicated sub-network to dynamically predict the pedestrian intention. The model combines the pedestrian motion state and scene information to dynamically adjust the end target of the pedestrian at different times. Qualitative experiments show that the end coordinates predicted by the model can gradually approach the real target, and the predicted trajectory is basically completely fitted with the real trajectory, which can effectively predict the deviation angle of the pedestrian movement and avoid collisions. Brief Description of the Drawings

[0022] Figure 1 is a flowchart of the pedestrian trajectory prediction method provided by the embodiment of the present invention;

[0023] Figure 2 is the flowchart of sub-network modeling in the method of the present invention;

[0024] Figure 3 is the schematic diagram of the combined modeling of the main network and the sub-network and trajectory decoding in the method of the present invention;

[0025] Figure 4 is the comparison chart of predicted trajectories in the experiment of the embodiment of the present invention. Detailed implementation manners

[0026] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0027] The present invention utilizes the characteristic that pedestrian intention information modeling can help improve the accuracy of trajectory prediction, and proposes a multi-task pedestrian trajectory prediction method based on a main network for pedestrian motion pattern modeling and a sub-network for dynamic analysis of pedestrian intentions. The present invention designs a multi-precision pedestrian feature representation method, uses a gated recurrent neural network (GRU) to encode the pedestrian's historical position information, dynamically analyzes the pedestrian's motion target through a target dynamic analysis sub-network, designs three sub-tasks to assist the backbone network in fitting the complete trajectory, and simultaneously uses an attention mechanism to fuse the model output vectors of the two networks. Finally, a conditional variational autoencoder (CVAE) is used to decode the encoded state to predict the future multi-modal trajectories of the pedestrian.

[0028] As Figure 1 shown, the pedestrian trajectory prediction method integrating motion target analysis in the embodiment of the present invention is described in the following four steps.

[0029] Step 1: Generate multi-precision data. Obtain the pedestrian's historical trajectory video, extract image frames from it, convert them into two-dimensional pedestrian position coordinates, and generate three data forms: the rough-precision coordinates of the pedestrian, the fine-precision features, and the dynamic scene information according to the pedestrian's position.

[0030] As Figure 1 shown, in the embodiment of the present invention, each observation time is 3.2 s, and 8 image frames are extracted at intervals of 0.4 s at each observation time to obtain the two-dimensional position coordinates of the pedestrian in each frame. The two-dimensional coordinates of the pedestrian are the fine-precision coordinates.

[0031] Step 1.1: Generate the rough-precision coordinates of the pedestrian.

[0032] The present invention divides the global scene into multiple sub-regions, and the regional coordinates of the scene are the input rough-precision coordinates, which retain the physical information of the scene position and are convenient for combination with the scene information. In the embodiment of the present invention, first, according to the coordinate size of the scene, the fine-precision distribution range of the scene is obtained (xmin , x max , y min , y max ), set the corresponding coarse - precision coordinate accuracy \(R = m\times n\), where both \(m\) and \(n\) are positive integers. Let the position coordinate of pedestrian \(i\) be \(p\) i , calculate the length and width of each coarse - precision area according to the distribution range of the scene and the set area accuracy, and then calculate the coarse - precision coordinate \(PR\) of the pedestrian according to the size of the coarse - precision. The detailed process can refer to Algorithm 1.

[0033]

[0034] As described in Algorithm 1, in the embodiment of the present invention, assume that there are \(N\) pedestrians in the scene. For each pedestrian \(i\), obtain the two - dimensional coordinates \((p\) i (x), p i (y)) in the scene, and then calculate the coarse - precision coordinate position \((PR\) i (x), PR i (y)) of each pedestrian \(i\) according to the set coarse - precision coordinate accuracy and the fine - precision distribution range of the scene. Here, \(i\) is the pedestrian number, taking values from 0 to \(N - 1\). The coarse - precision coordinates of \(N\) pedestrians in the scene form a matrix \(PR\). Obtain a pedestrian's coarse - precision coordinate for each frame of the image by executing Step 1.1.

[0035] Step 1.2: Pre - process the coarse - precision coordinates and fine - precision coordinates of the pedestrians, and construct coarse - precision information and fine - precision information respectively. The pre - processing includes relative coordinate transformation, calculation of speed and acceleration, etc.

[0036] Because the coordinate distribution ranges of different scenes vary greatly, in order to improve the migration ability of the model, the present invention sets the position of the last frame of the target pedestrian at the observation moment as the coordinate origin of the pedestrian, so that the coordinate distributions of different scenes can be controlled in a similar space. In the embodiment of the present invention, the last frame at an observation moment is the position \((x_8, y_8)\) at the 8th moment, which is used as the coordinate origin, and the absolute coordinates of the pedestrian position are transformed into relative coordinates as follows:

[0037] x t,new = x t - x_8 (1)

[0038] y t,new = y t - y_8 (2)

[0039] Among them, \((x\) t , y t ) is the pedestrian coordinate at time \(t\), \((x\) t,new , y t,new)It represents the relative coordinates transformed from the pedestrian's fine-precision coordinates at time t. Similarly, the relative coordinates of the pedestrian's coarse-precision coordinates can be transformed according to the position in the last frame of the observation time.

[0040] After transforming the coarse-precision coordinates and fine-precision coordinates of the pedestrian to obtain relative coordinates, in order to model the motion pattern of the pedestrian, it is also necessary to calculate the speed and acceleration, splice the position and speed features to form input data, and perform data augmentation to expand the training data set. In the present invention, the speed information and acceleration information of the pedestrian are calculated by using the methods of first-order derivative and second-order derivative. The three-dimensional features of coordinate position, speed, and acceleration are spliced together to jointly form the new input data of the model. And the method of trajectory rotation is also used for data augmentation. In the embodiment of the present invention, all the trajectories in the scene are rotated every 15 degrees to expand the training data set.

[0041] Step 1.3: Construct dynamic scene information.

[0042] The common methods for modeling scene information in existing methods mostly use the picture information of the scene or the semantic segmentation information of the scene picture to supplement the scene layout information. Although the semantic segmentation information has achieved good results in fields such as 3D stereo reconstruction, this semantic segmentation method has certain ambiguous information and lacks the interactive semantics between the scene and the pedestrian. For example, the lawn by the road and the lawn in the park have the same semantic segmentation meaning, but the lawn in the park generally allows pedestrians to move, while the lawn by the road generally prohibits pedestrians from moving. They have the same semantic information but have completely different movement rules. In order to solve the ambiguity problem of semantic segmentation for pedestrian movement, more effectively model the interactive semantics between scene information and pedestrian movement, and enable the scene information to more accurately guide the future movement of pedestrians, the method of the present invention obtains the probability density map passed by each scene position based on historical trajectory statistics, and provides the area distribution where pedestrians can move and the corresponding probability information.

[0043] Since the coarse-precision coordinates retain the spatial position information of the scene, the present invention uses the natural position information of the coarse-precision coordinates, combines the regional information of the coarse-precision coordinates with the local scene information of pedestrian movement, and further refines the regions divided by the coarse-precision coordinates. The coarse precision of the scene is R = m×n. In the embodiment of the present invention, each sub-region is further divided into fine regions with a precision of 9×9 to obtain the global scene information with a precision of R = 81×m×n. At each moment, the present invention dynamically models the local scene where the pedestrian is located according to the current coarse-precision coordinates of the pedestrian, guides the future movement of the pedestrian, and avoids the pedestrian moving to illegal areas. The dynamic modeling of the local scene where the pedestrian is located is described in step 2 below.

[0044] Step 2: Model the pedestrian motion pattern.

[0045] The present invention proposes a pedestrian trajectory prediction structure for joint main network - sub - network modeling, which divides pedestrian trajectory prediction into two parts: motion pattern modeling and dynamic analysis of pedestrian intention. A dedicated sub - network is used to dynamically predict the intention of pedestrians. Three sub - tasks are introduced in the dynamic prediction sub - network using a multi - precision data representation method. The multi - task approach is used to improve the model's ability to analyze pedestrian intention, and the attention mechanism method is used to fuse the main network and the sub - network to achieve synchronous update of the model. This step describes the pedestrian motion pattern modeling of the main network.

[0046] The backbone of the main network model is a GRU model, and the input of the model is the fine - precision coordinate information of pedestrians, which is used to model the motion sequence information of pedestrians. As shown in formula (3), the three input features of position, speed, and acceleration constructed in the previous step are encoded to obtain the vector e t . In addition to the motion state e t of pedestrians, as shown in formula (4), the method of the present invention also introduces the target intention vector g t of pedestrians, and combines the pedestrian motion state and target intention information to jointly encode the historical trajectory. At the end of each observation moment, the decoding module uses the multi - layer perceptron shown in formula (5) to map the output of the GRU to predict the end - point coordinates of pedestrians. The representation vector h t+1 of this target coordinate is fused with the output of the sub - network to generate a new target intention vector g t+1 , which, together with the hidden state h t+1 , is fed into the main network for the next - moment model iteration.

[0047] e t = f e (x t , y t , Δx t , Δy t , ax t , ay t ; W e ) (3)

[0048] h t = GRU(h t-1 , e t , g t-1 ; W GRU ) (4)

[0049]

[0050]

[0051] Among them, x t , y tis the position of the pedestrian at time t, Δx t , Δy t is the speed of the pedestrian at time t, ax t , ay t is the acceleration of the pedestrian at time t, f e is a multi-layer perceptron, W e is the model parameter of the multi-layer perceptron f e , e t is the vector representation after the model encodes the fine-precision motion state. h t is the motion state vector of the pedestrian at time t, g t is the intention vector of the target dynamics of the pedestrian at time t, and the h 1 and g 1 at the first moment are initialized as all-zero vectors, W GRU is the model weight of the GRU in the main network. f goal is a multi-layer perceptron, W goal is the model parameter of f goal , is the predicted end intention target of the pedestrian at time t by the model, p g is the true end coordinate of the pedestrian. Formula (6) is the loss function of the end decoding module in the main network, and MSE is used to calculate the mean square error.

[0052] Step 3: Pedestrian intention dynamic analysis.

[0053] In the main network, although the output h t of the GRU is used at each moment for pedestrian end target prediction, it is difficult to make the end prediction module of the model fully converge by simply using the mean square error function of Formula (6) as the loss function to update the model. In order to model the target intention of the pedestrian and make the model achieve a better convergence effect, the present invention designs a pedestrian target dynamic prediction sub-network to dynamically update the motion intention of the pedestrian.

[0054] As Figure 3 shown, the model input of the sub-network consists of the fine-precision coordinate information coarse-precision coordinate information of the pedestrian, and the scene information s t of the area where the pedestrian is located. First, the three input data of the model are encoded. Similar to Formula (3), the model uses a multi-layer perceptron to encode the fine-precision coordinate information and the coarse-precision coordinate information, and respectively obtains vectors and As shown in Formula (7), the model uses a convolutional neural network CNN to encode the local scene information s t to obtain

[0055]

[0056] Among them, W cnn is the model parameter of the convolutional neural network CNN.

[0057] In order to model the time series features and fuse them with the modeling information of the main network, the present invention also uses GRU to perform sequence modeling on the three types of information input to the sub-network. As shown in formula (8), the GRU model GRU sub of the sub-network has an input that includes three-dimensional information, as well as the intention prediction vector of the sub-network at the (t - 1)th moment W GRUsub is the training parameter of the model GRU sub .

[0058]

[0059] In order to model the regional features of the pedestrian's motion intention, in addition to predicting the fine-precision coordinates of the pedestrian's end moment, the present invention also proposes two additional sub-tasks to model the target region features of the pedestrian, namely, predicting the coarse-precision end coordinates of the pedestrian and scoring the intention end region of the pedestrian.

[0060] The first sub-task is shown in formula (9). The model uses a multi-layer perceptron f fine to map the pedestrian's motion intention vector to obtain the fine-precision coordinates of the pedestrian's end intention, where W fine is the update parameter of the model f fine .

[0061]

[0062] The second sub-task is shown in formula (10). The model uses a multi-layer perceptron f coarse to map the pedestrian's motion intention vector to obtain the coarse-precision coordinates of the pedestrian's end, where W coarse is the update parameter of the model f coarse .

[0063]

[0064] The third sub-task is shown in formula (11). First, use the matrix W score to Perform mapping, followed by the ReLU activation function, and then use the Softmax function to score the R = m×n sub-regions within the scene to obtain the scoring matrix score. The true value of each region's score is determined by the region where the pedestrian's end point is located. In the present invention, the score of the region where the pedestrian's end point is located is set to 1, and the scores of other sub-regions are set to 0.

[0065]

[0066] Among them, b score is a model parameter.

[0067] Through the above introduction, the loss function of the sub-network consists of three parts shown in formula (12), where is the fine-precision coordinates of the end point predicted by the model, p fine is the true fine-precision coordinates of the end point, p coarse is the coarse-precision coordinates of the end point predicted by the model, p coarse is the true coarse-precision coordinates of the end point, score is the region scoring result of the model, label is the true region scoring label, and L CE is the cross-entropy function. RMSE represents the calculation of the root mean square error.

[0068]

[0069] However, since the current sub-network and the main network are in a decoupled state, the target state vector of the sub-network cannot assist the main network in future trajectory prediction, nor can the loss function of the sub-network be used to assist the update of the main network model. In order to use the backpropagation of the model to synchronously update the two networks, the present invention designs two network fusion schemes for selection to couple the two parts of the network.

[0070] (1) The first network fusion method is to fuse the motion state of the main network with the important scene information screened by the sub-network. The sub-network of the model scores the importance of the m×n sub-regions at each moment, and selects the top K target sub-regions with the highest scores as the key regions. The selected K regions are encoded respectively using the CNN convolutional network shown in formula (7).

[0071]

[0072] After encoding the K regions, the weighted average of the encoded K scene information is obtained according to the importance scores using formula (13) to obtain the important region information that the pedestrian needs to consider Finally, the two networks are fused using the multi-head attention mechanism and residual connection shown in formulas (14) and (15) to obtain the target intention vector g t .

[0073]

[0074]

[0075] where <.,.> is the inner product operation, and W Q 、W K and W V are trainable parameter matrices, h t is the output vector of the main network GRU, D is the dimension number of h t , p is the number of heads of the multi-head attention mechanism, s r is the attention score, and g t is the target intention vector. denotes the r-th mapped vector obtained by mapping through the multi-head attention mechanism. denotes the r-th mapped vector obtained by mapping h t through the multi-head attention mechanism. The present invention uses the multi-head attention mechanism to map the vector h t , into multiple vectors, and then calculates the attention mechanism and integrates them respectively, which can capture feature information more fully.

[0076] (2) The fusion method introduced in the first method directly combines the scene information of K important regions, which may overly introduce artificially set rule information, and it is difficult to determine the optimal value of the parameter K. Therefore, the present invention attempts to directly fuse the GRU output of the sub-network with the GRU output h t of the main network using the attention mechanism introduced in formulas (14) and (15), that is, by replacing in the two formulas with , and then calculating the target intention vector g t . The structure for training the joint modeling of the main network and the sub-network is as Figure 3 shown.

[0077] Step 4: Future trajectory decoding.

[0078] As Figure 3 shown, in the trajectory decoding stage, the present invention uses the CVAE module to generate multi-modal trajectories. The CVAE module is divided into an encoding part and a decoding part. The encoding part is divided into an identification module q ψ (z|x,y) and a prior module p θ (z|x), where x represents the observed trajectory of the pedestrian, y represents the true future trajectory of the pedestrian, and z represents the high-dimensional latent code vector. q ψ (z|x,y) is to predict z by combining x and y, and p θ(z|x) predicts z based on x. In the present invention, the distributions of z predicted by these two models are made as similar as possible, so as to achieve the effect that the result predicted only using x is similar to the result predicted using x and y. During the test phase, only p θ (z|x) is used.

[0079] As shown in formula (16), at the last observation moment, the model encodes according to the historical motion state of the pedestrian to obtain the mean μ and variance σ corresponding to the Gaussian distribution, samples the Gaussian distribution to obtain the high-dimensional latent variable z ∼ N(μ, σ), and then uses formula (17) to splice the sampled high-dimensional latent variable z with the GRU output h of the main network obs to obtain the initial hidden state of the GRU The and the encoded current position of the pedestrian predicted by the model are jointly input into the GRU decoding module of formulas (18) and (19) to obtain the position vector at the next moment.

[0080] p(y|x) = p φ (y|z,x)p θ (z|x) (16)

[0081]

[0082]

[0083]

[0084] where f mlp , f pred , f decoder are all multi-layer perceptrons; represents the splicing operation; represents the initial vector input into the D-GRU network. D-GRU is the GRU in the decoding phase, and h obs represents the motion information of the pedestrian at time T<00, which is the motion information of the pedestrian at time T obs mlp 、W pred 、W decoder are the parameters of the models f mlp , f pred , f decoder respectively; represents the output of the GRU in the decoding phase.

[0085] During the test phase, in order not to introduce future information, the model directly samples the latent variable z from p θ (z|x) and will not introduce the recognition module q ψ(z|x,y). During the training phase, the present invention uses the KL (Kullback-Leibler Divergence) divergence to fit the predicted latent variable distribution to the true distribution, as shown in Equation (20).

[0086] Loss KLD = KLD(q ψ (z|x,y),p θ (z|x)) (20)

[0087] The present invention can jointly train the main network - sub-network joint pedestrian trajectory prediction and the CVAE model using the Variety Loss to generate multi-modal trajectories. The loss function is as shown in Equation (21) and consists of four parts, and the four-part loss function jointly updates the network.

[0088]

[0089] Where is the position of the k-th pedestrian trajectory in the multi-modal trajectory predicted by the model at time t, p t is the true position of the pedestrian at time t, Loss des is the loss function of the main network end point prediction module shown in Equation (6), Loss KLD is the CVAE module loss function shown in Equation (20), Loss sub is the sub-network loss function shown in Equation (12). T pre is the time to be predicted.

[0090] The implementation of the method of the present invention is as follows:

[0091] The experimental platform parameters are as follows:

[0092] CPU: Intel(R)Xeon(R)Gold 6226R CPU@2.90GHz

[0093] GPU: NVIDIA Tesla T4 GPU

[0094] Operating System: Ubuntu 18.04.6 LTS

[0095] Code running environment: Python 3.6.12, Pytorch 1.7.1

[0096] During the training process, the present invention uses the Adam optimizer for training. The batch size is set to 64, and the learning rate is 0.0001. The dimension of the hidden layer of the perceptron neurons is 256. In the rough-precision modeling, the model adopts different precision division strategies. In the small-scenario dataset of ETH-UCY, the division precision is a 5×5 area, and in the large-scenario dataset of Stanford Drone Dataset, the division precision is a 9×9 area. The scene information precision corresponding to each rough-precision area is 9×9. The dimension z of the latent variable sampled from the CVAE is 64, and the hyperparameter of the model Variety Loss is set to 20.

[0097] The present invention conducts experimental comparisons on public datasets. The results show that the results of the present invention are better than existing methods. The evaluation metrics are ADE / FDE. ADE is the sum of the squares of the distance differences between the predicted positions and the true positions for the next 12 frames (4.8 seconds), and FDE is the square of the distance difference between the predicted end position and the true end position. The smaller the ADE and FDE values, the better the effect.

[0098] Table 1. Comparison of test results of the present invention and other methods on public datasets

[0099] Model ETH HOTEL UNIV ZARA1 ZARA2 Average value PECNet 0.54 / 0.87 0.18 / 0.24 0.35 / 0.60 0.22 / 0.39 0.17 / 0.30 0.29 / 0.48 Trajectron++ 0.43 / 0.86 0.12 / 0.19 0.22 / 0.43 0.17 / 0.32 0.12 / 0.25 0.21 / 0.41 SGCN 0.63 / 1.03 0.32 / 0.55 0.37 / 0.70 0.29 / 0.53 0.25 / 0.45 0.37 / 0.65 The present invention 0.37 / 0.67 0.13 / 0.23 0.21 / 0.44 0.13 / 0.26 0.12 / 0.23 0.19 / 0.36

[0100] As can be seen from Table 1, the method of the present invention achieves the best effect in terms of the average value. The pedestrian trajectory prediction model of the present invention especially achieves good results in the two scenarios of ZARA1 and ZARA2. Through analysis, it is found that this is because the scenario is an open street scenario with many group interaction scenarios such as pedestrians walking in groups and multiple people staying and talking. The rough-precision coordinates and local scene information can model the regional information.

[0101] As Figure 4 shown, the present invention selects four groups of motion conditions on the public dataset to visually display the qualitative trajectory experiment of the model. Figure 4 It is divided into four scenarios. The solid line is the historical observation trajectory of the pedestrian, and the dotted line is the trajectory predicted by the model. Different pedestrian trajectories are predicted in the scenario. The present invention mainly selects two social motion patterns of group motion and pedestrian motion avoiding collision for display. In Figures A and B, multiple groups of pedestrians move in the same direction, and the results predicted by the model of the present invention are basically completely fitted with the true trajectories. In Figures C and D, the pedestrian motion trajectories deviate, and collisions with surrounding pedestrians and obstacles are avoided. The model of the present invention predicts the turning motion intention of the pedestrians and effectively predicts the deviation angle of the pedestrian motion to avoid collisions with vehicles and passing pedestrians.

[0102] Some existing methods directly use two-dimensional spatial coordinates as the intention information of pedestrians. However, the observable two-dimensional coordinates of the end point cannot fully describe the intention of pedestrians. Directly using the end point coordinates as the movement intention of pedestrians will make the model less robust. The method of the present invention proposes a dynamic analysis sub-network for pedestrian intention, which adopts a brand-new data representation method. The data of pedestrians is represented as three parts: fine-precision coordinates, coarse-precision coordinates, and dynamic scene information. And three sub-tasks are introduced to drive the main network to update together. While ensuring the trajectory prediction effect of the model, the analysis ability of pedestrian intention is improved and the robustness of the model is enhanced.

[0103] Except for the technical features described in the specification, the rest are known technologies to those skilled in the art. The present invention omits the description of well-known components and well-known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation manners described in the above embodiments do not represent all implementation manners consistent with the present application. Based on the technical solution of the present invention, various modifications or deformations that can be made by those skilled in the art without creative efforts are still within the protection scope of the present invention.

Claims

1. A pedestrian trajectory prediction method integrating moving target analysis, characterized in that, It includes the following steps: Step 1: Extract multi-precision data from the pedestrian historical trajectory video, including the coarse-precision coordinate information, fine-precision coordinate information, and dynamic scene information of the pedestrian; The coarse-precision coordinates are the sub-region coordinates where the pedestrian is located after dividing the scene area, and the fine-precision coordinates refer to the coordinates of the pedestrian in the scene; both the coarse-precision coordinate information and the fine-precision coordinate information include the pedestrian coordinate position, speed, and acceleration; Step 2: Construct a pedestrian trajectory prediction model that combines the main network and the sub-network; The sub-network of the pedestrian trajectory prediction model encodes three types of data: the input pedestrian rough-precision coordinate information, pedestrian fine-precision coordinate information, and dynamic scene information, and then uses the gated recurrent unit (GRU) neural network model sub for sequence modeling to predict and output the pedestrian motion intention vector at the current moment, and then inputs the pedestrian motion intention vector into three sub-task models; the first sub-task model uses a multi-layer perceptron f fine to map the pedestrian motion intention vector to obtain the fine-precision coordinates of the pedestrian's end point; the second sub-task model uses a multi-layer perceptron f coarse to map the pedestrian motion intention vector to obtain the rough-precision coordinates of the pedestrian's end point; the third sub-task model first uses a matrix W score to map the pedestrian motion intention vector, then connects the ReLU activation function, and then uses the Softmax function to score the importance of each sub-region in the scene; The main network of the pedestrian trajectory prediction model uses a multi-layer perceptron f e After encoding the input pedestrian fine-precision coordinate information, it is input into the gated recurrent neural network model GRU together with the target intention vector of the pedestrian at the current moment. The GRU outputs the pedestrian motion state vector at the current moment, and then uses the multi-layer perceptron f goal To map the output of the GRU and predict the pedestrian's end coordinates; the target intention vector of the pedestrian is obtained by fusing and calculating the output of the GRU of the sub-network sub And the output of the GRU of the main network using the attention mechanism; Step 3: In the trajectory decoding stage, use the conditional variational autoencoder CVAE to generate multi-modal trajectories.

2. The method according to claim 1, wherein In the above Step 1, the generation method of the coarse-precision coordinate information of the pedestrian is as follows: Divide the scene area according to the coarse precision of m×n, and calculate the coarse-precision coordinates of each pedestrian according to the coarse precision and the fine-precision coordinates; for each pedestrian, take the coordinate position in the last frame of the image at each observation moment as the origin, and convert the pedestrian coordinate positions in the extracted frames of the image at the observation moment into relative coordinate positions; then calculate the speed and acceleration of the pedestrian at each moment based on the relative coordinate positions; both m and n are positive integers.

3. The method according to claim 1, wherein In the above Step 2, encode the coarse-precision coordinate information and the fine-precision coordinate information of the pedestrian at time t respectively using a multi-layer perceptron, and encode the dynamic scene information using a convolutional neural network.

4. The method according to claim 1, wherein In the above Step 2, the target intention vector of the pedestrian is obtained through the following method: First, score the importance of each sub-region in the scene through a sub-network, select the top K sub-regions with the highest scores as key regions, and use a convolutional neural network to encode the K sub-regions; assume that for the multi-precision data processing at time t, the encoding vector of the dynamic scene information of the i-th sub-region selected is The importance score of the sub-region is score i , i = 1, 2, … K, then perform a weighted average on the K sub-region encodings according to the importance scores to obtain the key region information of the pedestrian Then, the outputs of the sub-network and the main network are fused using the multi-head attention mechanism and residual connections to obtain the target intention vector g of the pedestrian at time t t , as follows: where <.,.> is the inner product operation, and W Q , W K and W V are trainable parameter matrices, h t is the output vector of the main network GRU, D is the dimension number of h t , p is the number of heads of the multi-head attention mechanism, and s r is the attention score; is the r-th mapped vector obtained by mapping through the multi-head attention mechanism, is the r-th mapped vector obtained by mapping h t through the multi-head attention mechanism.

5. The method according to claim 1, wherein In the above Step 2, the target intention vector of the pedestrian is obtained through the following method: Suppose the pedestrian trajectory prediction model processes multi-precision data at time t to obtain the output of the GRU of the sub-network sub and the output of the GRU of the main network h t . The outputs of the sub-network and the main network are fused using the multi-head attention mechanism and residual connection to obtain the target intention vector g at the current time t t as follows:​ Among them, <.,.> is the inner product operation, W Q 、W K and W V is a trainable parameter matrix, h t is the output vector of the main network GRU, D is h t The number of dimensions, p is the number of heads of the multi-head attention mechanism, s r is the attention score; They are h t The rth mapping vector obtained by performing multi-head attention mechanism mapping.

6. The method according to claim 1, wherein In the said step 2, the fine-precision coordinate information (x t , y t , Δx t , Δy t , ax t , ay t ) of the pedestrian at time t is input into the main network. First, the multi-layer perceptron f e is used for encoding to obtain the vector e t . Then, combined with the target intention vector g t of the pedestrian at time t, they are jointly encoded through the GRU model to output the hidden state h t , which is expressed as: h t = GRU(h t-1 , e t , g t-1 ; W GRU ), where the h 1 and g 1 at the first moment have initial states as all-zero vectors, and W GRU is the model weight of the GRU in the main network; At the end point of each observation moment, the decoding module uses a multi-layer perceptron f goal to map the output of the GRU and predict the end point coordinates of the pedestrian which is expressed as: where W goal is the model parameter of f goal ​ 7. The method according to claim 1, wherein In step 2 described above, when training the pedestrian trajectory prediction model, the loss function Loss of the sub-network sub is as follows: Among them, RMSE represents the calculated root mean square error, and L CE is the cross-entropy function; is the predicted fine precision coordinate of the end point, and p fine is the true fine precision coordinate of the end point; is the predicted coarse precision coordinate of the end point, and p coarse is the true coarse precision coordinate of the end point; score is the regional importance scoring result output by the sub-network, label is the true regional scoring label, the score of the sub-region where the pedestrian end point is located is 1, and the scores of other sub-regions are 0; Loss function of the main network des As follows: Among them, is the predicted end target of the pedestrian at time t, and p g is the true end coordinate of the pedestrian, and MSE is the calculated mean square error.

8. The method according to claim 1, wherein In the aforementioned step 3, the CVAE is divided into an encoding part and a decoding part. The encoding part is divided into an identification module q ψ (z|x,y) and a prior module p θ (z|x), where x represents the observed trajectory of the pedestrian, y represents the true future trajectory of the pedestrian, and z is a high-dimensional latent variable; at the last observation moment, encoding is performed based on the historical motion state of the pedestrian to obtain the mean μ and variance σ corresponding to the Gaussian distribution, and sampling from the Gaussian distribution gives z ∼ N(μ, σ); Concatenate the sampled high-dimensional latent variable z with the GRU output h of the main network obs to obtain the initial hidden state of the GRU Let D-GRU denote the GRU in the decoding stage. Then, the output of the GRU in the decoding stage at time t+1 is as follows: Then predict the position of the pedestrian at the next moment as follows: Among them, f pred and f decoder are both multi-layer perceptrons, and W pred and W decoder are the parameters of f pred and f decoder respectively; is the predicted pedestrian position at time t, is the predicted pedestrian position at time t + 1; During the training phase, the KL divergence is used to fit the predicted high-dimensional latent variable distribution to the true distribution; during the test phase, the high-dimensional latent variable z is directly sampled from the prior module p θ (z|x).

9. The method according to claim 1 or 7 or 8, characterized in that In the said step 3, the pedestrian trajectory prediction model and the CVAE are jointly trained, and the loss function L used for training is as follows: variety as follows: Among them, T pre is the moment to be predicted, is the position of the k-th pedestrian trajectory to be predicted at time t, p t is the true position of the pedestrian at time t, Loss des is the loss function of the main network, Loss KLD is the loss function of the CVAE, Loss sub is the loss function of the sub-network; Use the trained model to predict the pedestrian trajectory.

Images