Pedestrian trajectory prediction method and device based on energy diffusion and symmetric network

Abstract

The application provides a pedestrian trajectory prediction method based on energy diffusion and symmetry network, relates to the field of pedestrian trajectory prediction, and comprises the following steps: constructing a symmetry pedestrian trajectory prediction network model; the symmetry pedestrian trajectory prediction network model comprises a global energy diffusion model, a scene heat map encoder, a trajectory encoder, a path node and target point decoder, and a trajectory decoder; obtaining a static scene pedestrian trajectory graph; the past behavior habit factors and future uncertainty factors of the pedestrian trajectory prediction are analyzed by the global energy diffusion model of the symmetry pedestrian trajectory prediction network model, so that the symmetry pedestrian trajectory prediction network model designed has interpretability; the multi-resolution feature map and the symmetry feature are input into the path node and target point decoder, the output of the path node and target point decoder is connected to the input of the trajectory encoder, and the global prediction capability of the model and the precision of the pedestrian trajectory prediction are improved.

Description

Technical Field

[0001] This application relates to the field of pedestrian trajectory prediction, and in particular to a method and device for pedestrian trajectory prediction based on energy diffusion and symmetric networks. Background Technology

[0002] Pedestrian trajectory prediction is a complex task that infers a pedestrian's future trajectory based on their historical movement patterns. To achieve this goal, researchers have designed various methods based on motion pattern analysis and learning of pedestrian behavior habits. For example, Yue et al. [1] A pedestrian trajectory prediction model was designed to explain pedestrian movement behavior based on the social relevance of the pedestrian's surroundings. Wang et al. [2] A recurrent network is proposed for estimating pedestrian trajectory habits. Additionally, Fourier transform and spectral analysis are also used to obtain pedestrian trajectory habits. [3] Since pedestrians are usually in groups, some studies have used energy diffusion-based methods to explore the correlation between adjacent particles in order to facilitate the study of commonalities in the movement patterns of individuals within the group. [4] These methods analyze an individual's behavioral trajectories within a crowd, simulating their waypoints as individuals with coherent movement patterns. However, existing technologies suffer from the following two drawbacks:

[0003] First, the prediction model is not global enough, resulting in low long-term prediction accuracy. Existing methods mainly model pedestrians' historical movement habits, but behavioral habits can only reflect the past movement trajectory trends of pedestrians and cannot cope with future uncertainties and random factors.

[0004] Second, the deep learning networks used in the prediction algorithms lack interpretability. Due to the strong unpredictability and multiple prediction possibilities in pedestrian trajectory prediction, existing methods lack a theoretical approach to explain this unpredictability and the interaction between individual pedestrians and surrounding groups in a moving scene, resulting in a lack of interpretability in the prediction results. Summary of the Invention

[0005] The purpose of this application is to provide a pedestrian trajectory prediction method and device based on energy diffusion and symmetric networks, which can improve the accuracy and interpretability of pedestrian trajectory prediction, in order to solve the problems mentioned in the background art above.

[0006] The above-mentioned objective of this application is achieved through the following technical solution:

[0007] S1: Construct a symmetric pedestrian trajectory prediction network model, which includes: a global energy diffusion model, a scene heat map encoder, a trajectory encoder, a path node and target point decoder, and a trajectory decoder;

[0008] S2: Obtain the static scene pedestrian trajectory map and perform preprocessing to obtain the scene heat map and pedestrian motion semantic segmentation map;

[0009] S3: Analyze the scene heatmap using a global energy diffusion model to determine influencing factors and trajectory heatmaps;

[0010] S4: Encode the scene heatmap using a scene heatmap encoder to determine the scene segmentation multi-resolution feature map;

[0011] S5: Encode the tensors of the trajectory heatmap and scene heatmap using a trajectory encoder to determine the multi-resolution feature map for trajectory prediction;

[0012] S6: Construct multi-resolution feature maps and symmetric features; determine the heatmaps of path nodes and target points based on the path node and target point decoders, multi-resolution feature maps, and symmetric features;

[0013] S7: Input the path node and target point heatmap and trajectory heatmap into the trajectory encoder, and the trajectory decoder outputs the predicted future trajectory map of the pedestrian.

[0014] Optionally, step S2 includes: using a pre-trained semantic segmentation model to perform semantic segmentation of the road layout on the static scene pedestrian trajectory map, and combining it with a global energy diffusion model to obtain a scene heatmap and a pedestrian motion semantic segmentation map G. I Pedestrian motion semantic segmentation map G I This includes: standing semantics, walking semantics, and running semantics.

[0015] Optionally, step S3 includes:

[0016] The global energy diffusion model is as follows:

[0017] Define the adjacency similarity S between nodes i and j of adjacent routes of a trajectory. t (i,j), as follows:

[0018] S t (i,j)=max(R t (i,j),0)

[0019] Among them, R t (i,j) represents the correlation between the movement speeds of a pedestrian at nodes i and j at time t.

[0020] Using a path η of length h h ={w0→w1→…→w h}, w0→w1→…→w h This represents the trajectory points of a path from node w to node j, used to define path η. hOverall probability as follows:

[0021] Among them, w b This represents the b-th trajectory point;

[0022] Since there are multiple paths between node i and node j, we define the path probability g of path length h between node i and node j. h (i,j) represents the following:

[0023]

[0024] Among them, g h This represents the probability between pairs of nodes, measured by the multiple possibilities existing in a pedestrian's past movement habits; assuming the set of all paths between node i and node j that require passing through h neighboring nodes is ψ. h η h ∈ψ h This indicates one of the paths;

[0025] Considering the motion state of node i as the heat source particle of the thermal energy diffusion field, and the motion state of node j as the adjacent particle influenced by the previous node through energy diffusion, the continuously diffused energy influences the entire trajectory point group until a steady state is reached. Combining the thermodynamic diffusion model, we have the following expression:

[0026]

[0027] in, Heat source particles The thermal energy diffused at node i after t seconds, m p It is the propagation coefficient; It is the heat source particle P i The initial motion optical flow before time zero They represent particles P and P respectively. i The velocity components along the horizontal x-axis and vertical y-axis in a two-dimensional image;

[0028] The instant of energy diffusion between particles is regarded as time zero. The total response of the signal and the system is used as the energy transfer principle. The initial energy of the particle itself before time zero is defined as the zero-input response, and the energy brought by the signal input to the particle at time zero is defined as the zero-state response. The influencing factors are determined. Therefore, in terms of pedestrian trajectory prediction, the starting time of the predicted pedestrian movement is taken as time zero. The influencing factors include: the zero-input response corresponds to the past behavioral habits of pedestrians before time zero, and the zero-state response corresponds to the future uncertainty of pedestrian trajectory prediction.

[0029] make Let R represent the two-dimensional coordinates of the pedestrian at time k in the heatmap, and let R represent the set of real numbers, thus obtaining the pedestrian's historical movement trajectory. Where k p Represents the past k p The number of video frames; k represents the sequence variable, from the first frame to the kth frame. p Frames, total k p Frames serve as a sequence of historical motion trajectories;

[0030] Based on the historical movement trajectories of pedestrians in the static scene trajectory map, the trajectory heatmap Q(k,i,j) of scene heatmap I is calculated using the following formula:

[0031]

[0032] Where (i,j) represents the position coordinates in the heatmap, and (x,y) represents the two-dimensional coordinates of a point in the scene heatmap I.

[0033] Optionally, step S5 includes:

[0034] The trajectory encoder is an independent encoder, including N E Each downsampling block is used; upsampling is performed in each downsampling block using a 2×2 convolution and the ReLU activation function.

[0035] Process the tensor Q of the scene heatmap I using a trajectory encoder. G Determine the multi-resolution feature map for trajectory prediction.

[0036] Optionally, step S6 includes:

[0037] S61: Introduce the extension arm of the symmetric pedestrian trajectory prediction network model to connect the resolution levels of the scene segmentation multi-resolution feature map and the trajectory prediction multi-resolution feature map, and construct a multi-resolution feature map.

[0038] S62: Merge symmetrical features of different resolutions in the scene heatmap encoder and the path node and target point decoder;

[0039] S63: Based on the path node and target point decoders, multi-resolution feature maps, and symmetric features, the output probability distribution estimation results are as follows:

[0040] The probability matrix P represents the predicted overall future position distribution of pedestrians through the path node and target point decoders; the softmax function is used to represent the probability distribution estimation result of the overall future position (i,j) of pedestrians.

[0041]

[0042] Among them, P ijThis represents the probability that the pedestrian's overall future position will be in (i,j);

[0043] S64: Use the pixel-by-pixel sigmoid activation function to convert the probability distribution estimation results into the form of a heatmap, and determine the heatmaps of path nodes and target points.

[0044] Optionally, step S7 further includes: integrating different semantic branches of the pedestrian motion semantic segmentation map and past trajectory heatmaps to collect information on trajectory nodes and target points and input it into the trajectory encoder.

[0045] An electronic device includes a processor, a memory, a user interface, and a network interface. The memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to enable the electronic device to perform a pedestrian trajectory prediction method based on energy diffusion and symmetric networks.

[0046] A computer-readable storage medium storing instructions that, when executed, perform a pedestrian trajectory prediction method based on energy diffusion and symmetric networks.

[0047] The beneficial effects of the technical solution provided in this application are:

[0048] 1. The symmetric pedestrian trajectory prediction network model consists of a pair of symmetric U-Net networks, which symmetrically distribute semantic segmentation and trajectory prediction, thereby improving the global prediction capability of the symmetric pedestrian trajectory prediction network model.

[0049] 2. Based on the principle of global energy diffusion, a global energy diffusion model is constructed, analyzing the mutual influence between energy transfer and the relationship between particles. Pedestrian trajectory nodes are abstracted into particle energy diffusion representations, thus providing an effective theoretical analysis of the correlation between pedestrian trajectory nodes. A novel interpretable theoretical framework is proposed, which, by combining the zero-input response and zero-state response of the signal and the total response of the system, reveals the factors of past movement habits and future movement uncertainties in human trajectory prediction, providing interpretability for the proposed symmetric pedestrian trajectory prediction network model. Attached Figure Description

[0050] The present application will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0051] Figure 1 This is a flowchart of the pedestrian trajectory prediction algorithm based on energy diffusion and symmetric networks in the embodiments of this application.

[0052] Figure 2This is an error comparison diagram of the pedestrian trajectory prediction method based on energy diffusion and symmetric network in the embodiments of this application;

[0053] Figure 3 This is a schematic diagram of the electronic device structure of the pedestrian trajectory prediction method based on energy diffusion and symmetric networks in the embodiments of this application. Detailed Implementation

[0054] To provide a clearer understanding of the technical features, objectives, and effects of this application, the specific embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0055] Embodiments of this application provide a pedestrian trajectory prediction method based on energy diffusion and symmetric networks.

[0056] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of a pedestrian trajectory prediction method based on energy diffusion and symmetric networks in an embodiment of this application, including:

[0057] S1: Construct a symmetric pedestrian trajectory prediction network model, which includes: a global energy diffusion model, a scene heat map encoder, a trajectory encoder, a path node and target point decoder, and a trajectory decoder;

[0058] S2: Obtain the static scene pedestrian trajectory map and perform preprocessing to obtain the scene heat map and pedestrian motion semantic segmentation map;

[0059] S3: Analyze the scene heatmap using a global energy diffusion model to determine influencing factors and trajectory heatmaps;

[0060] S4: Encode the scene heatmap using a scene heatmap encoder to determine the scene segmentation multi-resolution feature map;

[0061] S5: Encode the tensors of the trajectory heatmap and scene heatmap using a trajectory encoder to determine the multi-resolution feature map for trajectory prediction;

[0062] S6: Construct multi-resolution feature maps and symmetric features; determine the heatmaps of path nodes and target points based on the path node and target point decoders, multi-resolution feature maps, and symmetric features;

[0063] S7: Input the path node and target point heatmap and trajectory heatmap into the trajectory encoder, and the trajectory decoder outputs the predicted future trajectory map of the pedestrian.

[0064] Specifically, existing research mainly focuses on predicting pedestrians' past movement habits, failing to model the uncertainties of future pedestrian trajectories globally, resulting in significant unpredictability in the prediction models. Therefore, improving the efficiency of model input information utilization and the interpretability of the models themselves is urgently needed. This paper establishes a global energy diffusion model by considering the interactions between pedestrians and surrounding pedestrians in a crowd, studying past movement habits and future movement uncertainties. While existing technologies only consider the current input signal, this application constructs a global energy diffusion system using the full response, considering both the zero-state response generated by the current input signal and the zero-input response generated by the initial signal before the current moment. Through global energy diffusion analysis, both past movement habits at the current moment and subsequent future movement uncertainties are incorporated into the trajectory prediction. By modeling the zero-input response and zero-state response separately, the paper analyzes the past movement habits and future movement uncertainties of pedestrians in a group environment, thus providing a more global explanation of the different influencing factors in pedestrian trajectory prediction.

[0065] Step S2 includes: using a pre-trained semantic segmentation model to perform semantic segmentation of the road layout on the static scene pedestrian trajectory map, and combining it with a global energy diffusion model to obtain a scene heatmap and a pedestrian motion semantic segmentation map G. I Pedestrian motion semantic segmentation map G I This includes: standing semantics, walking semantics, and running semantics.

[0066] Step S3 includes:

[0067] The global energy diffusion model is as follows:

[0068] Define the adjacency similarity S between nodes i and j of adjacent routes of a trajectory. t (i,j), as follows:

[0069] S t (i,j)=max(R t (i,j),0)

[0070] Among them, R t (i,j) represents the correlation between the movement speeds of a pedestrian at nodes i and j at time t.

[0071] Using a path η of length h h ={w0→w1→…→w h}, w0→w1→…→w h This represents the trajectory points of a path from node i to node j, used to define path η. h Overall probability as follows:

[0072] Among them, w b This represents the b-th trajectory point;

[0073] Since there are multiple paths between node i and node j, we define the path probability g of the path with length h between node i and node h. h (i,h) represents the following:

[0074]

[0075] Among them, g h This represents the probability between pairs of nodes, measured by the multiple possibilities existing in a pedestrian's past movement habits; assuming the set of all paths between node i and node j that require passing through h neighboring nodes is ψ. h η h ∈ψ h This indicates one of the paths;

[0076] Considering the motion state of node i as the heat source particle of the thermal energy diffusion field, and the motion state of node j as the adjacent particle influenced by the previous node through energy diffusion, the continuously diffused energy influences the entire trajectory point group until a steady state is reached. Combining the thermodynamic diffusion model, we have the following expression:

[0077]

[0078] in, Heat source particles The thermal energy diffused at node i after t seconds, m p It is the propagation coefficient; It is the heat source particle P i The initial motion optical flow before time zero They represent particles P and P respectively. i The velocity components along the horizontal x-axis and vertical y-axis in a two-dimensional image;

[0079] The instant of energy diffusion between particles is regarded as time zero. The total response of the signal and the system is used as the energy transfer principle. The initial energy of the particle itself before time zero is defined as the zero-input response, and the energy brought by the signal input to the particle at time zero is defined as the zero-state response. The influencing factors are determined. Therefore, in terms of pedestrian trajectory prediction, the starting time of the predicted pedestrian movement is taken as time zero. The influencing factors include: the zero-input response corresponds to the past behavioral habits of pedestrians before time zero, and the zero-state response corresponds to the future uncertainty of pedestrian trajectory prediction.

[0080] make Let R represent the two-dimensional coordinates of the pedestrian at time k in the heatmap, and let R represent the set of real numbers, thus obtaining the pedestrian's historical movement trajectory. Where kp Represents the past k p The number of video frames; k represents the sequence variable, from the first frame to the kth frame. p Frames, total k p Frames serve as a sequence of historical motion trajectories;

[0081] Based on the historical movement trajectories of pedestrians in the static scene trajectory map, the trajectory heatmap Q(k,i,j) of scene heatmap I is calculated using the following formula:

[0082]

[0083] Where (i,j) represents the position coordinates in the heatmap, and (x,y) represents the two-dimensional coordinates of a point in the scene heatmap I.

[0084] Specifically, suppose the set of all paths between node i and node j that require passing through h neighboring nodes is ψ. h η h ∈ψ h It is one of the paths, g ηh (i,j) represents the result after η h The path similarity between node i and node j is calculated using the formula... The similarity of all adjacent nodes along the path is calculated by sequentially multiplying each pair of nodes. This represents the sum of the similarities of all paths of length h between node i and node j. This similarity is the probability of a path of length h.

[0085] Step S5 includes:

[0086] The trajectory encoder is an independent encoder, including N E Each downsampling block is used; upsampling is performed in each downsampling block using a 2×2 convolution and the ReLU activation function.

[0087] Process the tensor W of the scene heatmap I using a trajectory encoder. G Determine the multi-resolution feature map for trajectory prediction.

[0088] Specifically, in order for the trajectory module of the trajectory encoder to form a symmetrical structure with the trajectory decoder, the tensor W of the observed scene heatmap... G This application designs an additional U-Net encoder (track encoder) instead of using a shared U-Net encoder, such as... Figure 1 As shown, this forms the left half of an H-shaped symmetrical pedestrian trajectory prediction network model. Here, each U-Net encoder consists of N... E The system consists of several downsampling blocks, each with corresponding max pooling and progressively increasing channel depth. In this way, the final deep learning feature representation Q...E and N E The intermediate feature tensors of layer -1 form a compact space, providing diverse spatial resolutions for the subsequent trajectory target decoder, thereby predicting the future k. f Pedestrian positions at any given time

[0089] Specifically, the input of the trajectory encoder in this application is connected to the output of the path node and target point decoder. The path node and target point decoder propagate useful semantic information to the simplified trajectory module (i.e., the trajectory module in the trajectory encoder and trajectory decoder), which then assists the trajectory module in focusing more on areas that may be the target and ignoring other unimportant areas. Because pedestrians are usually drawn to the trajectory target and tend to walk towards that destination, the additional semantic information is used to guide the generated trajectory towards the target through target traction. The semantic information is the output of the path node and target point decoder.

[0090] Step S6 includes:

[0091] S61: Introduce the extension arm of the symmetric pedestrian trajectory prediction network model to connect the resolution levels of the scene segmentation multi-resolution feature map and the trajectory prediction multi-resolution feature map, and construct a multi-resolution feature map.

[0092] S62: Merge symmetrical features of different resolutions in the scene heatmap encoder and the path node and target point decoder;

[0093] S63: Based on the path node and target point decoders, multi-resolution feature maps, and symmetric features, the output probability distribution estimation results are as follows:

[0094] The probability matrix P represents the predicted overall future position distribution of pedestrians through the path node and target point decoders; the softmax function is used to represent the probability distribution estimation result of the overall future position (i,j) of pedestrians.

[0095]

[0096] Among them, P ij This represents the probability that the pedestrian's overall future position will be in (i,j);

[0097] S64: Use the pixel-by-pixel sigmoid activation function to convert the probability distribution estimation results into the form of a heatmap, and determine the heatmaps of path nodes and target points.

[0098] Specifically, the path node and target point decoders aim to provide possible clues about pedestrian motion targets and traversed waypoints for trajectory prediction, thus forming a symmetrical relationship with the scene encoder. The two encoders generate two independent encoded features for scene segmentation and trajectory prediction, which are equally important; therefore, these two independent encoded features are combined as the input terminals of the path node and target point decoders. Extended arms of the U-Net network structure, which introduces a symmetric pedestrian trajectory prediction network model, connect different resolution levels. The multi-resolution feature maps are upsampled in each block using 2×2 convolutions and the ReLU activation function, a process called deconvolution. Unlike the 3×3 convolutions used in the encoder, 2×2 convolutions double the number of channels. In particular, the path node and target point decoders increase the size of the multi-resolution feature maps from H... E ×W E The result is H×W. While increasing the resolution by reducing the number of channels through convolution, the resolution of the multi-resolution feature map is also increased. Because the size of the feature map is increased, some feature details are inevitably lost during the encoder's downsampling operation. Symmetrical features from different resolutions in the encoder and decoder are merged to complete the resolution details.

[0099] Step S7 includes: integrating different semantic branches of the pedestrian motion semantic segmentation map and past trajectory heatmaps to collect information on trajectory nodes and target points and input it into the trajectory encoder.

[0100] Specifically, an additional extension arm input is added between the decoder and encoder, namely the outputs of the path node and target point decoders. Following the previous approach, the node heatmaps required by different decoder layers are downsampled to match the magnitude of the input information. After the final pixel-by-pixel sigmoid processing, a probability estimate of the pedestrian's trajectory in the next n seconds is obtained, where n = k. f / T fps T fps For frame rate.

[0101] Specifically, the pedestrian trajectory prediction method in this application has higher prediction accuracy and can better provide probability density estimates of future pedestrian trajectories. This application uses the proposed H-Net model (symmetric pedestrian trajectory prediction network model), which outperforms existing models in both ADE and FDE. By introducing global energy diffusion analysis based on zero input and zero state response, the learning mechanism of the model for input trajectory data is elucidated, improving the efficiency of utilizing observation data information. Furthermore, this approach also significantly improves the interpretability of the model, providing a new approach for trustworthy AI (artificial intelligence technology).

[0102] Table 1

[0103]

[0104] Table 1 shows the experimental results of the prediction method of this application and existing methods on the SDD pedestrian trajectory prediction dataset. Compared with existing models, the experimental results of this application outperform existing methods in the test results on the real dataset, as detailed below:

[0105] Test Dataset Overview: The Stanford Drone Dataset (SDD) comprises over 11,000 individual pedestrians across 20 top-down scenes observed using a flying drone over the Stanford University campus. It includes over 40,000 pedestrian-scene interactions and has been widely used in trajectory prediction literature for short-term horizon settings. For short-term prediction, to demonstrate consistency with other methods, a common pedestrian trajectory prediction setting is used for comparison. This model achieves this in T... fps When the value is 2.5, sampling yields a length of k. p =8 (3.2 seconds) input sequence and length k f =12 (4.8 seconds) of output sequence.

[0106] Evaluation Metrics: In the pedestrian trajectory prediction experiments, this application primarily uses two common pedestrian trajectory error metrics, ADE and FDE, to measure the accuracy of the algorithm. ADE is the average distance error, representing the average error between the predicted path and the actual path over a future period. FDE is the final distance error, representing the path distance error at the end of the prediction period; both have different practical meanings.

[0107] In the experiment, the segmented scene map and the preprocessed pedestrian historical motion trajectory were fed into two encoders of H-Net, and the final output results were compared with the true values ​​in the dataset to verify the effectiveness of this application. Analysis of the results shows that the proposed global energy diffusion analysis theory significantly improves prediction accuracy. As can be seen from the experimental results in Table 1, the ADE and FDE of the H-Net model outperform other models on the SDD dataset.

[0108] Figure 2 This is a comparison chart of the errors of the prediction method proposed in this application with other mainstream methods under different prediction duration settings.

[0109] Furthermore, this application also includes a prediction comparison experiment with different durations ranging from 5 seconds to 60 seconds. The experimental results are as follows: Figure 2 As shown in the experimental results of long-term prediction, the model of this application outperforms other mainstream comparative models in both ADE and FDE indices. Its advantage increases with the length of prediction time. The long-term prediction experiment further verifies the effectiveness of this application.

[0110] [1]Jiangbei Yue,Dinesh Manocha,and He Wang.Human trajectoryprediction via neural social physics.In Computer Vision-ECCV 2022:17thEuropean Conference,Tel Aviv,Israel,October 23-27,2022,Proceedings,PartXXXIV,pages 376-394.Springer,2022.

[0111] [2]Chuhua Wang,Yuchen Wang,Mingze Xu,and David J Crandall.Stepwisegoal-driven networks for trajectory prediction.IEEE Robotics and AutomationLetters,7(2):2716-2723,2022.

[0112] [3]Conghao Wong,Beihao Xia,Ziming Hong,Qinmu Peng,Wei Yuan,Qiong Cao,Yibo Yang,and Xinge You.View vertically:A hierarchical network for trajectoryprediction via fourier spectrums.In Computer Vision-ECCV 2022:17th EuropeanConference,Tel Aviv,Israel,October 23-27,2022,Proceedings,Part XXII,pages682-700.Springer,2022.

[0113] [4]Andreas Lugmayr,Martin Danelljan,Andres Romero,Fisher Yu,RaduTimofte,and Luc Van Gool.Repaint:Inpainting using denoising diffusionprobabilistic models.In Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition,pages11461-11471,2022.

[0114] [5]Agrim Gupta,Justin Johnson,Li Fei-Fei,Silvio Savarese,andAlexandre Alahi.Social gan:Socially acceptable trajectories with generativeadversarial networks.In Proceedings of the IEEE conference on computer visionand pattern recognition,pages 2255-2264,2018.

[0115] [6]Apratim Bhattacharyya,Michael Hanselmann,Mario Fritz,BerntSchiele,and Christoph-Nikolas Straehle.Conditional flow variationalautoencoders for structured sequence prediction.arXiv preprint arXiv:1908.09008,2019.

[0116] [7]Nachiket Deo and Mohan M Trivedi.Trajectory forecasts in unknownenvironments conditioned on grid-based plans.arXiv preprint arXiv:2001.00735,2020.

[0117] [8] Junwei Liang, Lu Jiang, and Alexander Hauptmann. Simaug: Learningrobust representations from 3d simulation for pedestrian trajectory prediction in unseen cameras. arXiv preprint arXiv:2004.02022,2,2020.

[0118] [9]Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, and Adrien Gaidon. It is not the journey but the destination: Endpoint conditioned trajectory prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28,2020,Proceedings,Part II 16,pages759-776.Springer,2020.

[0119] It should be noted that the above embodiments of the apparatus are only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0120] This application also discloses an electronic device. (See reference...) Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. The electronic device 500 may include: at least one processor 501, at least one network interface 504, a user interface 503, a memory 505, and at least one communication bus 502.

[0121] The communication bus 502 is used to enable communication between these components.

[0122] The user interface 503 may include a display screen and a camera. Optionally, the user interface 503 may also include a standard wired interface and a wireless interface.

[0123] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0124] The processor 501 may include one or more processing cores. The processor 501 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 505, and by calling data stored in memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.

[0125] The memory 505 may include random access memory (RAM) or read-only memory.

[0126] Optionally, the memory 505 includes a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. The memory 505 may also optionally include, but is not limited to, at least one storage device located remotely from the aforementioned processor 501. (Refer to...) Figure 3 The memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for predicting pedestrian trajectories based on energy diffusion and symmetric networks.

[0127] exist Figure 3 In the illustrated electronic device 500, the user interface 503 is mainly used to provide an input interface for the user and acquire user input data; while the processor 501 can be used to call an application program stored in the memory 505 for predicting pedestrian trajectories based on energy diffusion and symmetric networks. When executed by one or more processors 501, the electronic device 500 performs one or more methods as described in the above embodiments. It should be noted that, for the foregoing method embodiments, for the sake of simplicity, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0128] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0129] In the various embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed include, but are not limited to, indirect couplings or communication connections between apparatuses or units through some service interfaces, including but not limited to electrical or other forms.

[0130] The units described as separate components include, but are not limited to, physically separate units. The components shown as units include, but are not limited to, physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of these units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or may exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0133] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0134] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. A pedestrian trajectory prediction method based on energy diffusion and symmetric networks, characterized in that, The method includes the following steps: S1: Construct a symmetric pedestrian trajectory prediction network model, which includes: a global energy diffusion model, a scene heat map encoder, a trajectory encoder, a path node and target point decoder, and a trajectory decoder; S2: Obtain the static scene pedestrian trajectory map and perform preprocessing to obtain the scene heat map and pedestrian motion semantic segmentation map; S3: Analyze the scene heatmap using a global energy diffusion model to determine influencing factors and trajectory heatmaps; Step S3 includes: The global energy diffusion model is as follows: Define the nodes of adjacent routes of a trajectory. With nodes Adjacency similarity ,as follows: in, yes Pedestrians at the node and nodes The correlation of movement speed; With a length of path , Represents a node To the node The trajectory points of a path are used to define the path. Overall probability ,as follows: in, Indicates the first A trajectory point; Due to the node and nodes There are multiple paths between them, therefore a node is defined. and nodes The path length between them is Path probability , means as follows: in, This represents the probability between pairs of nodes, measured by the multiple possibilities existing based on a pedestrian's past movement habits; assuming... Nodes and The nodes need to go through The set of all paths to n adjacent nodes is , This indicates one of the paths; Node The motion state is regarded as the heat source particle of the thermal energy diffusion field, and the node is considered as the heat source particle of the thermal energy diffusion field. The motion state is considered as the influence of the neighboring particles after the previous node affects them through energy diffusion. The continuously diffused energy affects the entire group of trajectory points and eventually reaches a steady state. Combining the thermodynamic diffusion model, we have the following expression: in, Heat source particles At the node Passing through The heat energy after the diffusion in seconds, It is the propagation coefficient; Heat source particles The initial motion optical flow before time zero They represent particles respectively The velocity components along the horizontal x-axis and vertical y-axis in a two-dimensional image; The instant of energy diffusion between particles is regarded as time zero. The total response of the signal and the system is used as the energy transfer principle. The initial energy of the particle itself before time zero is defined as the zero-input response, and the energy brought by the signal input to the particle at time zero is defined as the zero-state response. The influencing factors are determined. Therefore, in terms of pedestrian trajectory prediction, the starting time of the predicted pedestrian movement is taken as time zero. The influencing factors include: the zero-input response corresponds to the past behavioral habits of pedestrians before time zero, and the zero-state response corresponds to the future uncertainty of pedestrian trajectory prediction. make For pedestrians in the heat map The two-dimensional coordinate position at time [time]. Represent the set of real numbers to obtain the historical movement trajectory of pedestrians. ,in Indicating the past One video frame; Represents a sequence variable, from the first frame in the past to the last frame. Frames, total Frames serve as a sequence of historical motion trajectories; Calculate the scene heatmap based on the historical movement trajectories of pedestrians in the static scene. trajectory heatmap The formula is as follows: in Indicates the position coordinates on the heat map. Representing a scene heatmap The two-dimensional coordinates of a point in the data; S4: Encode the scene heatmap using a scene heatmap encoder to determine the scene segmentation multi-resolution feature map; S5: Encode the tensors of the trajectory heatmap and scene heatmap using a trajectory encoder to determine the multi-resolution feature map for trajectory prediction; S6: Construct multi-resolution feature maps and symmetric features; determine the heatmaps of path nodes and target points based on the path node and target point decoders, multi-resolution feature maps, and symmetric features; S7: Input the path node and target point heatmap and trajectory heatmap into the trajectory encoder, and the trajectory decoder outputs the predicted future trajectory map of the pedestrian.

2. The pedestrian trajectory prediction method based on energy diffusion and symmetric networks as described in claim 1, characterized in that, Step S2 includes: using a pre-trained semantic segmentation model to perform semantic segmentation of the road layout on the static scene pedestrian trajectory map, and combining it with a global energy diffusion model to obtain a scene heatmap and a pedestrian motion semantic segmentation map. pedestrian motion semantic segmentation map This includes: standing semantics, walking semantics, and running semantics.

3. The pedestrian trajectory prediction method based on energy diffusion and symmetric networks as described in claim 1, characterized in that, Step S5 includes: The trajectory encoder is an independent encoder, including... Each downsampling block is used; downsampling is performed using a 2×2 convolution and the ReLU activation function in each downsampling block. Use a trajectory encoder to process scene heatmaps tensor Determine the multi-resolution feature map for trajectory prediction.

4. The pedestrian trajectory prediction method based on energy diffusion and symmetric networks as described in claim 1, characterized in that, Step S6 includes: S61: Introduce the extension arm of the symmetric pedestrian trajectory prediction network model to connect the resolution levels of the scene segmentation multi-resolution feature map and the trajectory prediction multi-resolution feature map, and construct a multi-resolution feature map. S62: Merge symmetrical features of different resolutions in the scene heatmap encoder and the path node and target point decoder; S63: Based on the path node and target point decoders, multi-resolution feature maps, and symmetric features, the output probability distribution estimation results are as follows: The probability matrix P represents the overall future location distribution of pedestrians predicted by the path node and target point decoders; using The function represents the pedestrian's overall future position. The probability distribution estimation results are as follows: in, Indicates the pedestrian's overall future location The probability of; S64: Use the pixel-by-pixel sigmoid activation function to convert the probability distribution estimation results into the form of a heatmap, and determine the heatmaps of path nodes and target points.

5. The pedestrian trajectory prediction method based on energy diffusion and symmetric networks as described in claim 1, characterized in that, Step S7 further includes: integrating different semantic branches of the pedestrian motion semantic segmentation map and past trajectory heatmaps to collect information on trajectory nodes and target points and input it into the trajectory encoder.

6. An electronic device, characterized in that, The device includes a processor (501), a memory (505), a user interface (503), and a network interface (504). The memory (505) is used to store instructions. The user interface (503) and the network interface (504) are used to communicate with other devices. The processor (501) is used to execute the instructions stored in the memory (505) to cause the electronic device to perform the method as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed, perform the steps of the method as described in any one of claims 1-5.

Images