Pedestrian trajectory prediction method and device based on scene information fusion and electronic equipment

Abstract

The embodiment of the invention provides a pedestrian trajectory prediction method and device based on scene information fusion and electronic equipment. The pedestrian trajectory prediction method comprises the steps of obtaining pedestrian trajectory data and scene data in a current environment; determining trajectory fusion data based on the pedestrian trajectory data and the scene data, wherein the trajectory fusion data comprises influence information of different scenes on the pedestrian trajectory; processing the trajectory fusion data based on a convolutional neural network and a graph attention network to obtain trajectory global interaction data, the trajectory global interaction data including influence information of different pedestrians on pedestrian trajectories; and predicting the trajectory of each pedestrian in the current environment in a future preset time period based on the trajectory global interaction data. According to the embodiment of the invention, a scene attention mechanism is introduced, and the influence of different scenes on the pedestrian trajectory is combined, so that the accuracy of pedestrian trajectory prediction is effectively improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving, specifically to a method, device, and electronic device for predicting pedestrian trajectories based on scene information fusion. Background Technology

[0002] In intelligent driving and intelligent transportation systems, pedestrian trajectory prediction is one of the key technologies for achieving safe and efficient path planning. Autonomous vehicles need to possess predictive capabilities similar to human drivers, able to anticipate pedestrian movement trends and make appropriate avoidance and decision-making in complex traffic environments. If a vehicle cannot accurately predict pedestrian trajectories, it may lead to emergency braking, insufficient avoidance, or collision risks, thereby affecting driving safety.

[0003] In real-world environments, pedestrian movement patterns are influenced not only by neighboring pedestrians but also by the surrounding environment, such as pedestrians waiting at traffic lights, walking along sidewalks, or navigating around obstacles. These factors determine the rationality and feasibility of pedestrian trajectories. Therefore, a lack of modeling of global pedestrian-scene interactions may cause predicted trajectories to deviate from reasonable walking paths, thereby affecting the decision-making quality of autonomous vehicles. Summary of the Invention

[0004] In view of this, this application provides a method, device and electronic device for predicting pedestrian trajectories based on scene information fusion, so as to solve the problem of low accuracy of pedestrian trajectory prediction in the prior art.

[0005] In a first aspect, embodiments of this application provide a pedestrian trajectory prediction method based on scene information fusion, including: Acquire pedestrian trajectory data and scene data in the current environment; Trajectory fusion data is determined based on the pedestrian trajectory data and the scene data, and the trajectory fusion data includes information on the impact of different scenes on pedestrian trajectories; The trajectory fusion data is processed using convolutional neural networks and graph attention networks to obtain global trajectory interaction data, which includes information on the influence of different pedestrians on pedestrian trajectories. Based on the global trajectory interaction data, the trajectories of each pedestrian in the current environment are predicted within a future preset time period.

[0006] In one optional embodiment, determining the trajectory fusion data based on the pedestrian trajectory data and the scene data includes: Extract trajectory features from the pedestrian trajectory data and scene features from the scene data; A weighted scene feature vector is determined based on the trajectory features and the scene features. The weighted scene feature vector is used to indicate the degree of influence of different scenes on pedestrian trajectories. The pedestrian trajectory data and the weighted scene feature vector are concatenated to obtain the trajectory fusion data.

[0007] In one optional embodiment, determining the weighted scene feature vector based on the trajectory features and the scene features includes: The trajectory features are then subjected to dimensionality-upgrading processing based on the first linear transformation layer to obtain the trajectory feature vector. The scene features are reduced in dimensionality using the second linear transformation layer to obtain the scene feature vector. The trajectory feature vector and the scene feature vector are concatenated to obtain a fused feature vector; The fused feature vector is processed using a multilayer perceptron and the Softmax activation function to obtain an attention weight vector; The scene features are processed based on the attention weight vector to obtain the weighted scene feature vector.

[0008] In one optional embodiment, the processing of the trajectory fusion data based on convolutional neural networks and graph attention networks to obtain global trajectory interaction data includes: The trajectory fusion data is processed based on the convolutional neural network to obtain local trajectory interaction data; The local interaction data of the trajectory is processed based on the graph attention network to obtain the global interaction data of the trajectory.

[0009] In one optional embodiment, processing the trajectory fusion data based on the convolutional neural network to obtain trajectory local interaction data includes: Based on N The M-convolutional kernel performs feature capture on the trajectory fusion data; The feature data captured by the convolution kernel is nonlinearly activated based on the parameterized modified linear unit PReLU.

[0010] In one optional embodiment, processing the local trajectory interaction data based on the graph attention network to obtain global trajectory interaction data includes: The nodes and spatiotemporal edges of the graph attention network are constructed based on the local interaction data of the trajectory. The dynamic attention weights between nodes are determined based on the nodes and the spatiotemporal edges. Based on the nodes and their corresponding dynamic attention weights in the graph attention network, the nodes are aggregated to obtain the global interaction data of the trajectory.

[0011] In one optional embodiment, the step of predicting the trajectories of pedestrians in the current environment within a preset future time period based on the global trajectory interaction data includes: The spatial features of the trajectory global interaction data are extracted based on a multi-layer convolutional network decoder, and each convolutional layer of the multi-layer convolutional network decoder is connected to a parameterized corrected linear unit (PReLU). Based on the extracted spatial features, the trajectory probability distribution of each pedestrian within a preset time period is output. The trajectory probability distribution is used to indicate the probability value of the pedestrian moving in different directions. The direction with the highest probability value is determined as the pedestrian's target trajectory direction.

[0012] In one optional embodiment, processing the scene features based on the attention weight vector to obtain the weighted scene feature vector includes: For any scene dimension, the value of each network is multiplied by its corresponding attention weight and summed to obtain the weighted scene feature vector. The matrix format of the weighted scene feature vector is as follows: 1 C.

[0013] Secondly, embodiments of this application provide a pedestrian trajectory prediction device based on scene information fusion, comprising: The acquisition module is used to acquire pedestrian trajectory data and scene data in the current environment; The determination module is used to determine trajectory fusion data based on the pedestrian trajectory data and the scene data, wherein the trajectory fusion data contains information on the impact of different scenes on pedestrian trajectories; The processing module is used to process the trajectory fusion data based on convolutional neural networks and graph attention networks to obtain global trajectory interaction data, which includes information on the influence of different pedestrians on pedestrian trajectories. The prediction module is used to predict the trajectory of each pedestrian in the current environment within a preset time period in the future, based on the global trajectory interaction data.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to execute the method described in any of the first aspects above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the method described in any of the first aspects.

[0016] Fifthly, embodiments of this application provide a computer program product comprising executable instructions that, when executed on a computer, cause the computer to perform the method described in any of the first aspects.

[0017] The solution provided in this application involves acquiring pedestrian trajectory data and scene data in the current environment; determining trajectory fusion data based on the pedestrian trajectory data and the scene data, wherein the trajectory fusion data contains information on the influence of different scenes on pedestrian trajectories; processing the trajectory fusion data using a convolutional neural network and a graph attention network to obtain global trajectory interaction data, which contains information on the influence of different pedestrians on pedestrian trajectories; and predicting the trajectories of each pedestrian in the current environment within a preset future time period based on the global trajectory interaction data. This application introduces a scene attention mechanism, combining the influence of different scenes on pedestrian trajectories, effectively improving the accuracy of pedestrian trajectory prediction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A flowchart illustrating a pedestrian trajectory prediction method based on scene information fusion provided in this application embodiment. Figure 1 ; Figure 2 A flowchart illustrating another pedestrian trajectory prediction method based on scene information fusion provided in this application embodiment. Figure 2 ; Figure 3 This is an example schematic diagram of scene attention information fusion provided in an embodiment of this application; Figure 4 This is an example schematic diagram illustrating how an encoder structure and a graph attention network jointly generate hidden states for global interactions, as provided in an embodiment of this application. Figure 5 This is a schematic diagram of an example graph attention network provided in an embodiment of this application; Figure 6 A schematic diagram of a network structure for pedestrian trajectory prediction provided in an embodiment of this application; Figure 7 A schematic diagram of the structure of a pedestrian trajectory prediction device based on scene information fusion provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To better understand the technical solution of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0021] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0022] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0023] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0024] Currently, pedestrian trajectory prediction typically only considers local interactions, neglecting global scene information. In real-world environments, pedestrian movement patterns are influenced not only by neighboring pedestrians but also by the surrounding environment, such as pedestrians waiting at traffic lights, walking along sidewalks, or navigating obstacles. These factors determine the rationality and feasibility of pedestrian trajectories. Therefore, a lack of modeling for global pedestrian-scene interactions may cause predicted trajectories to deviate from reasonable walking paths, thus affecting the decision-making quality of autonomous vehicles.

[0025] To address the aforementioned issues, this application provides a pedestrian trajectory prediction method based on scene information fusion. By combining scene information with global pedestrian interaction information, the accuracy of pedestrian trajectory prediction is improved.

[0026] Figure 1 A flowchart illustrating a pedestrian trajectory prediction method based on scene information fusion provided in this application embodiment. Figure 1 This method can be applied to vehicle systems, such as... Figure 1 As shown, the method may include: Step 101: Obtain pedestrian trajectory data and scene data in the current environment.

[0027] During vehicle operation, onboard sensors can collect real-time information about the vehicle's environment to assist driving decisions. Optionally, onboard sensors may include onboard cameras and vehicle radar. Onboard cameras can collect scene data of the vehicle's environment, such as road structure, surrounding facilities, and road obstacles. Vehicle radar can collect pedestrian trajectory data around the vehicle; for example, it can collect the trajectory coordinates of pedestrians within 3.2 seconds, with a sampling interval of 0.4 seconds. The collected trajectory coordinates can be stitched together in chronological order to obtain pedestrian trajectory data.

[0028] Step 102: Determine trajectory fusion data based on pedestrian trajectory data and scene data. The trajectory fusion data contains information on the impact of different scenes on pedestrian trajectories.

[0029] Information on the impact of different scenarios on pedestrian trajectories can be used to characterize the importance of different scenarios in predicting pedestrian trajectories. For example, when a pedestrian is about to cross an intersection, the zebra crossing and traffic lights have a more critical impact on the pedestrian trajectory.

[0030] Optionally, the process of determining trajectory fusion data by the vehicle system may include: (1) extracting trajectory features from pedestrian trajectory data and scene features from scene data; (2) determining a weighted scene feature vector based on trajectory features and scene features, the weighted scene feature vector being used to indicate the degree of influence of different scenes on pedestrian trajectories; and (3) concatenating pedestrian trajectory data and weighted scene feature vector to obtain trajectory fusion data.

[0031] In (1), the vehicle system can extract trajectory features from pedestrian trajectory data, which can be output in vector form. Similarly, the vehicle system can also extract scene features from scene data, which can also be output in vector form.

[0032] In (2), the vehicle system concatenates and fuses the vector-form trajectory features and scene features to obtain a fused feature vector. Optionally, if the trajectory features and scene features have different vector dimensions, the vehicle system can first perform dimensionality-upgrading processing on the trajectory features based on the first linear transformation layer to obtain the trajectory feature vector, and then perform dimensionality-reducing processing on the scene features based on the second linear transformation layer to obtain the scene feature vector. If the trajectory feature vector and scene feature vector have the same dimension, the vehicle system concatenates the two to obtain a fused feature vector.

[0033] The vehicle system then processes the fused feature vector using a multilayer perceptron (MLP) and the softmax activation function to obtain an attention weight vector. This attention weight vector can be used to represent different scene features, and by processing scene features based on the attention weight vector, a weighted scene feature vector can be obtained.

[0034] In one optional embodiment, the matrix format of the attention weight vector includes: (H W), the matrix format of scene features includes: (H) W C), H, and W are used to determine the grid coordinates in the scene image. Each scene image contains H. W grids and C scene dimensions. For any scene dimension, the vehicle system multiplies the value of each network by its corresponding attention weight and sums the results to obtain a weighted scene feature vector. The matrix format of the weighted scene feature vector is: 1 C. The numerical value of this weighted scene feature vector can be used to indicate the impact information (i.e., the importance) of different scenes on pedestrian trajectories.

[0035] For example, in a scene of pedestrians crossing the road, the scene image is divided into 7 7-grid, each network contains 512-dimensional visual semantic features (such as zebra crossing texture, vehicle lanes, or traffic light signals), and the attention weight vector is a 7-dimensional grid. A 7-matrix structure. For any scene dimension, multiplying the value of each network in the scene image by the value at the corresponding matrix position in the attention weight vector and summing the results yields a 1-matrix structure. A 512-dimensional weighted scene feature vector. Each value in this weighted scene feature vector represents the importance of scene dimensions (such as zebra crossings and traffic lights) to pedestrian trajectory prediction.

[0036] In (3), the vehicle system concatenates the weighted scene feature vector with the original pedestrian trajectory data (i.e. trajectory coordinates) to obtain trajectory fusion data, so that each trajectory coordinate is expanded from the original (x, y) form to the (x, y, S) form, where S can represent the weighted scene feature vector.

[0037] Step 103: The trajectory fusion data is processed based on convolutional neural networks and graph attention networks to obtain global trajectory interaction data, which contains information on the influence of different pedestrians on pedestrian trajectories.

[0038] Global trajectory interaction data can be used to represent the influence of pedestrians on pedestrian trajectories, such as brief avoidance behaviors between pedestrians, following behaviors between pedestrians, or acceleration / deceleration behaviors of pedestrians themselves. Optionally, the vehicle system can first process the trajectory fusion data based on a convolutional neural network to obtain local trajectory interaction data. Then, it can process the local trajectory interaction data based on a graph attention network to obtain global trajectory interaction data. The influence information of different pedestrians on pedestrian trajectories may specifically include the dynamic attention weights of the graph attention network described below.

[0039] In determining the local interaction data of the trajectory, the vehicle system can first base it on N. The M-series convolutional kernel captures features from the trajectory fusion data, and then performs non-linear activation processing on the captured features using the parameterized modified linear unit (PReLU). Specifically, the convolutional neural network extracts local features from temporal information by sliding each feature channel along the time dimension; its kernel size can be set to 3. 3. One step at a time. This convolutional operation effectively captures the characteristics of pedestrian trajectory features changing over time across different channels. A PReLU layer is then added after the convolutional neural network to enhance the model's expressive power. The formula for the local interaction data of the trajectory is expressed as follows:

[0040] in, For convolutional networks, For PReLU function, and They are and The weight, For weighted scene feature vectors, This is pedestrian trajectory data (the sequence of trajectory coordinates of pedestrian i within the time window [1,T]). For local interaction intermediate features, Enhanced features for local interactions (enhanced the nonlinear representation of local patterns).

[0041] In practical processing, the vehicle system can iterate the above process multiple times to obtain local trajectory interaction data, which is then used as nodes in a graph attention network. The relationships between the nodes in the graph attention network can be modeled to obtain global trajectory interaction data. Specifically, the vehicle system first constructs the nodes and spatiotemporal edges of the graph attention network based on the local trajectory interaction data, and then determines the dynamic attention weights between nodes based on the nodes and spatiotemporal edges. For temporal edges: determine "which historical time steps have a greater impact on the current pedestrian trajectory" (e.g., nodes in the most recent second have higher weights). For spatial edges: determine "which pedestrians have a greater impact on the current pedestrian trajectory" (e.g., pedestrians who are close or have conflicting directions have higher weights). Then, based on the nodes in the graph attention network and their corresponding dynamic attention weights, the nodes are aggregated to obtain the global trajectory interaction data. The calculation formula for the global trajectory interaction data is expressed as follows:

[0042] in, For the weights of the graph attention network, It represents the interaction characteristics between nodes i and j (indicating the interaction strength between pedestrian i and neighbor j at time t). It is the aggregated hidden state of pedestrian i at time t, which contains global interaction information from other pedestrians. It is a nonlinear function.

[0043] Step 104: Based on the global trajectory interaction data, predict the trajectory of each pedestrian in the current environment within a preset time period in the future.

[0044] The vehicle system processes global trajectory interaction data using a convolutional network decoder and Parametric Corrected Linear Unit (PReLU), outputting the trajectory probability distribution of each pedestrian within a preset future time period. This trajectory probability distribution indicates the probability of a pedestrian moving in different directions. The decoder consists of seven convolutional layers, each followed by a PReLU activation function. In this decoder, the convolutional layers progressively extract spatial features from the input global trajectory interaction data and enhance non-linear expressiveness through the PReLU activation function, thus addressing the "dead neuron" problem that ReLU might introduce. Each convolutional operation captures deeper local patterns, while PReLU ensures the network can still effectively learn features even in negative regions. Through seven convolutional layers, the decoder can progressively extract and reconstruct features from low-level to higher-level abstract features, ultimately achieving accurate prediction of pedestrian trajectories.

[0045] In the decoder of this convolutional neural network, the output is designed as a Gaussian distribution, including the mean, variance, and correlation coefficient in the x and y directions. The mean represents the expected position of the predicted trajectory, the variance reflects the uncertainty of the prediction, and the correlation coefficient describes the correlation between the trajectories in the x and y directions. To obtain the most probable trajectory, the model calculates its probability density in the given Gaussian distribution and selects the trajectory with the highest probability as the final result, which effectively improves the accuracy and stability of the prediction.

[0046] In one optional embodiment, negative log-likelihood (NLL) can be used to optimize the model parameters during decoder training. After the decoder outputs a Gaussian probability distribution, the testing device calculates the NLL loss, which measures the difference between the predicted probability distribution and the true location. By iteratively optimizing the decoder parameters using the descent gradient of the NLL loss, the trajectory prediction becomes more closely aligned with the real-world scenario.

[0047] This application's embodiments introduce a scene attention mechanism, enabling the model to more accurately focus on the dynamic changes of key areas in the environment, effectively improving prediction accuracy. Simultaneously, by combining a convolutional neural network encoder (CNN), the model can extract deep-level feature information from complex scenes, enhancing its spatial information representation capabilities. Furthermore, parallel computing accelerates the model's computation process, making the processing of large-scale data more efficient. Building upon this, a graph attention network further optimizes information transmission between nodes, enhancing the model's ability to model interactions between pedestrians. In addition, the model employs a Gaussian distribution output, which helps capture the uncertainty of trajectory prediction, providing more flexible and reliable prediction results. Finally, the negative log-likelihood (NLL) loss function optimizes the training process, improving prediction stability and robustness. Overall, this invention demonstrates higher accuracy and stronger generalization ability when handling trajectory prediction tasks in complex scenes.

[0048] The above process will be described below through specific embodiments, with reference to... Figure 2 The method may include: Step 201: Obtain historical pedestrian trajectory data and scene data.

[0049] The vehicle's onboard cameras capture images of the surrounding environment in real time, allowing the vehicle system to obtain scene data. The vehicle's radar collects the movement trajectories of pedestrians in real time, and the vehicle system can obtain the coordinates (x, y) of a pedestrian every 0.4 seconds over the past 3.2 seconds, for a total of 8 points.

[0050] Step 202, scene feature extraction.

[0051] The pre-trained VGG-19 network is used to process scene data captured by the camera. The last fully connected layer (classification layer) is removed, and the feature maps output by the convolutional layers are retained. The output is a scene feature map with 512 channels (e.g., assuming the original image is 224x224, after passing through the VGG convolutional layers, it might be a 7x7x512 feature map). This feature map encodes the visual semantic information of the scene (such as roads, sidewalks, buildings, vehicles, and the positions of other pedestrians). This scene feature map provides the model with rich environmental contextual information. For example, the scene feature map extracted by VGG-19 will have different activation patterns (high values) in the "pedestrian crossing" area, the "bus stop" area on the roadside, and the "traffic flow" area in the middle of the road. This information is crucial for predicting whether pedestrians will cross the road or stop at the bus stop.

[0052] Step 203, trajectory feature extraction.

[0053] The target pedestrian's coordinate sequence over the past 8 time steps (t-7, t-6, ..., t-0) is: [(x7, y7), (x6, y6), ..., (x0, y0)]. The vehicle system concatenates the 16 values ​​(8 x + 8 y) from these 8 points in chronological order into a vector of length 16.

[0054] Step 204: Scene attention information fusion.

[0055] Trajectory Feature Upscaling: A linear transformation layer (fully connected layer) maps the low-dimensional (16-dimensional) trajectory feature vector to a high-dimensional feature vector (e.g., 256-dimensional) to capture more complex motion patterns (such as acceleration, deceleration, and turning tendency). The output is a high-dimensional feature vector T_high (256-dimensional) representing the historical trajectory.

[0056] Scene feature dimensionality reduction: Directly inputting high-dimensional scene feature maps into subsequent networks results in enormous computational costs and redundancy. Vehicle systems can use another linear transformation layer (fully connected layer) to reduce the dimensionality (e.g., to 256 dimensions). The output is the dimensionality-reduced scene feature vector S_low (e.g., 256 dimensions), preserving the most important scene information.

[0057] Feature concatenation: The vehicle system concatenates T_high (256-dimensional), representing trajectory features, and S_low (256-dimensional), representing scene features, to form a fused feature vector F_fused (512-dimensional). This fused feature vector combines dynamic (trajectory) and static (scene) information.

[0058] Determine the weighted scene feature vector: Input the fused feature vector F_fused into a multilayer perceptron (MLP), which outputs the degree of influence of different scenes on the future trajectory of pedestrians. Pass the output of the MLP through a softmax function to generate an attention weight vector A. A[i] represents the influence strength (weight) of the i-th scene on the predicted pedestrian trajectory, and the sum of all A[i] is 1. Use A to perform a weighted summation on the original, undimension-reduced scene feature map (512 channels) (essentially a spatial weighted pooling). Specifically, for each channel, calculate the weighted average of all spatial location feature values, and finally obtain a weighted scene feature vector S_att (512 dimensions). For example, for a pedestrian walking towards a crosswalk, the weights A of the "crosswalk" region and the "opposite sidewalk" region in the scene will be high, while the weights of "cars in the middle of the road" or "buildings in the distance" will be low. S_att contains the most important scene region information for predicting pedestrian trajectory (such as the direction and location of the crosswalk).

[0059] Determine the trajectory fusion data: For the coordinates (xt, yt) of each time step (t-7, t-6, ..., t-0) in the historical pedestrian trajectory data, concatenate the weighted scene feature vector S_att (511 dimensions) with it to output a new sequence, namely the trajectory fusion data. The trajectory fusion data contains the degree of influence of each scene on the future pedestrian trajectory. For example, at time t-4 (when the pedestrian has just stepped onto the crosswalk), its features not only include the position (x4, y4), but also the key scene information S_att about the crosswalk ahead and the opposite sidewalk, greatly enriching the information content of each historical point.

[0060] The steps for scene attention information fusion can be referred to Figure 3 The process is as follows: Scene data is input into a Convolutional Neural Network (CNN), a deep learning model specifically designed for processing data with a grid structure (such as images). Convolutional layers extract scene features. The CNN inputs these scene features into a fully connected layer A (similar to the second linear transformation layer mentioned above). Layer A performs dimensionality reduction on the scene features, resulting in a scene feature vector. Pedestrian trajectory data has its trajectory features extracted and then input into a fully connected layer B (similar to the first linear transformation layer mentioned above). Layer B performs dimensionality increase on the trajectory features, resulting in a trajectory feature vector. The outputs of fully connected layers A and B are concatenated to form a fused feature vector. This fused feature vector is then processed through a multilayer perceptron and activation functions to obtain a weighted scene feature vector. This weighted scene feature vector is then concatenated with the pedestrian trajectory data to obtain the fused trajectory data.

[0061] Step 205: Temporal feature extraction and global interaction modeling.

[0062] Determining Local Trajectory Interaction Data: Feature extraction is performed on the trajectory fusion data using one or more 1D convolutional layers (Conv1D) with a kernel size of 3 (spanning 3 adjacent time steps) and a stride of 1. The convolutional kernel slides independently on each feature channel (i.e., convolution across the time dimension). For example, a 3x1 convolutional kernel operates on the value of a channel across 3 consecutive time steps. The convolutional layer captures the change patterns of trajectory features within a local time window (e.g., velocity changes, directional fine-tuning), and then introduces non-linearity through the PReLU activation function. Two such convolutional neural network (Conv1D + PReLU) modules are stacked. The first module processes the original fusion sequence and outputs a new sequence representation (the dimension may change). The second module then processes this new sequence to further extract more abstract local temporal patterns, ultimately obtaining the trajectory local interaction data H_local.

[0063] Determine the global trajectory interaction data: Treat the feature vector of each time step in the local trajectory interaction data H_local as a node in a graph attention network. Aggregate the nodes through the graph attention network to obtain the global trajectory interaction data H_global, which contains the global dependencies between nodes. For example, the node at time t-0 (the most recent time) can highly focus on the node at time t-7 (the earliest time). If the pedestrian has been steadily moving in the same direction since t-7, this long-term consistency is important for predicting the future. At the same time, the node at time t-4 (the key decision point) will receive high attention weight because it marks the beginning of the pedestrian crossing the road.

[0064] Figure 4 This diagram illustrates how the encoder structure and the graph attention network jointly generate the hidden states of global interactions. Pedestrian trajectory data is processed through 2D convolution (Convolution2–Dimensional, Conv2D) and parametric rectified linear units (PReLU) to obtain local trajectory interaction data. The global trajectory interaction data is then fed into the graph attention network to obtain the global trajectory interaction data. a, b, and c represent different boundaries, and Wab, Wac, and Wbc represent the dynamic attention weights between different nodes.

[0065] Figure 5 The diagram shows the structure of a graph attention network, where h1, h2, h3, h4, and h5 are different nodes. In this diagram, h1 is the target node, and h2, h3, h4, and h5 are the neighboring nodes of h1. These are the dynamic attention weights between h1 and each node.

[0066] Step 206, decoder trajectory prediction.

[0067] The vehicle system uses a decoder consisting of seven 1D convolutional layers (Conv1D), with each convolutional layer followed by a PReLU activation function. The decoder ultimately outputs a sequence whose length should be consistent with the length of the future trajectory to be predicted (e.g., predicting the future in 3.2 seconds, with 0.4-second intervals, would output 8 points).

[0068] Step 207: Probabilistic output of pedestrian trajectory.

[0069] After trajectory prediction, the decoder outputs Gaussian distribution parameters. For each future time step (t+1, t+2, ..., t+8) in the predicted sequence, it outputs a 5-dimensional vector [μ_x, μ_y, σ_x, σ_y, ρ]. μ_x and μ_y are the mean values ​​of the predicted position in the x and y directions, respectively, representing the most likely expected position. σ_x and σ_y are the standard deviations of the predicted position in the x and y directions, representing the range of uncertainty in the prediction. ρ is the correlation coefficient between the x and y coordinates (between -1 and 1), used to describe the correlation between the two directions of movement (e.g., a positive correlation indicates a tendency to move towards the northeast or southwest simultaneously).

[0070] Figure 6 This diagram illustrates the network structure for pedestrian trajectory prediction. Scene data, after being processed by models such as CNNs, is combined with pedestrian trajectory data to obtain a weighted scene feature vector. The weighted scene feature vector is then concatenated with the pedestrian trajectory data to obtain trajectory fusion data. This fusion data is passed through an encoder to obtain global trajectory interaction data. Finally, a decoder processes this global trajectory interaction data to predict pedestrian trajectories. The encoder is composed of a graph attention network and multiple Conv2D and PReLU algorithms, while the decoder is also composed of multiple Conv2D and PReLU algorithms.

[0071] The pedestrian trajectory prediction method of this application embodiment was verified based on actual data. Five public datasets (eth, hotel, univ, zara1, zara2) containing pedestrian movement trajectories in different environments were divided into three parts: training set, test set, and validation set. The experiment used a training batch size of 64 and trained the model for 400 epochs. The learning rate was 0.001, decreasing to 0.5 times the original value every 50 epochs. The evaluation metrics for the prediction results were the Average Displacement Error (ADE) and the Final Displacement Error (FDE), representing the mean Euclidean distance between all nodes in the predicted trajectory and the true trajectory, respectively, and the Euclidean distance between the last position in the predicted trajectory and the true trajectory. The prediction time was 4.8 seconds. The experimental results are shown in Table 1.

[0072] Table 1 compares the relevant metrics (ADE / FDE) of the existing model and the model of the embodiment of this application under five datasets. The last row (Ours) contains the data for the model in this application embodiment, and the last column (Avg) represents the mean of the five data sets. It can be seen that the model in this application embodiment has the smallest average ADE and FDE errors across the five datasets. Furthermore, the parameter count (Params(K)) for each model is shown in Table 2, with the last row (SACPred(Ours)) representing the parameter count of the model in this application embodiment. It can be seen that although the parameter count of the model in this application embodiment is 19% and 7.9% higher than Trajectron++ and ScePT, respectively, ADE is improved by 62.5% compared to Trajectron++, and FDE is improved by 61.8% and 71.2% compared to Trajectron++ and ScePT, respectively. This demonstrates that the model in this application embodiment can maintain a relatively small parameter count while significantly improving accuracy.

[0073]

[0074] Table 2. Comparison of model parameter counts In this embodiment, scene attention information is fused with historical pedestrian trajectories before predicting future pedestrian trajectories, which can effectively improve the accuracy of future pedestrian trajectory prediction, thereby more accurately assisting autonomous driving and improving the safety of autonomous driving.

[0075] Figure 7 This is a schematic diagram of a pedestrian trajectory prediction device based on scene information fusion, provided as an embodiment of this application. Figure 7 As shown, the device may include: The acquisition module 710 is used to acquire pedestrian trajectory data and scene data in the current environment.

[0076] The determination module 720 is used to determine trajectory fusion data based on pedestrian trajectory data and scene data. The trajectory fusion data contains information on the impact of different scenes on pedestrian trajectories.

[0077] The processing module 730 is used to process the trajectory fusion data based on the convolutional neural network and the graph attention network to obtain global trajectory interaction data, which contains information on the influence of different pedestrians on the pedestrian trajectory.

[0078] The prediction module 740 is used to predict the trajectory of each pedestrian in the current environment within a preset time period in the future, based on the global trajectory interaction data.

[0079] Corresponding to the above embodiments, this application also provides an electronic device. Figure 8This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device 800 may include a processor 801, a memory 802, and a communication unit 803. These components communicate through one or more buses. Those skilled in the art will understand that the structure of the electronic device shown in the figure does not constitute a limitation on the embodiment of this application. It may be a bus-shaped structure or a star-shaped structure, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0080] The communication unit 803 is used to establish a communication channel, enabling the electronic device to communicate with other devices. It receives user data sent by other devices or sends user data to other devices.

[0081] The processor 801 serves as the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs, instructions, and / or modules stored in the memory 802, and calls data stored in the memory to perform various functions and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 801 may consist only of a central processing unit (CPU). In this embodiment, the CPU may have a single processing core or include multiple processing cores.

[0082] The memory 802 is used to store the execution instructions of the processor 801. The memory 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0083] When the execution instructions in memory 802 are executed by processor 801, the electronic device 800 is able to perform some or all of the steps in the above embodiments.

[0084] In a specific implementation, this application also provides a computer storage medium, which may store a program. When the program is executed, it may include some or all of the steps in the various embodiments of the pedestrian trajectory prediction method based on scene information fusion provided in this application. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0085] In a specific implementation, this application also provides a computer program product, wherein the computer program product includes executable instructions, which, when executed on a computer, cause the computer to perform some or all of the steps in various embodiments of the pedestrian trajectory prediction method based on scene information fusion provided in this application.

[0086] This application also provides a non-transitory computer-readable storage medium that stores computer instructions that cause the computer to execute the pedestrian trajectory prediction method based on scene information fusion provided in this application.

[0087] The aforementioned non-transitory computer-readable storage medium may be any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory, optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in connection with an instruction execution system, apparatus, or device.

[0088] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0089] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0090] Those skilled in the art will clearly understand that the techniques in the embodiments of this application can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of this application, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application or some parts of the embodiments.

[0091] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments and terminal embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

Claims

1. A pedestrian trajectory prediction method based on scene information fusion, characterized in that, include: Acquire pedestrian trajectory data and scene data in the current environment; Trajectory fusion data is determined based on the pedestrian trajectory data and the scene data, and the trajectory fusion data includes information on the impact of different scenes on pedestrian trajectories; The trajectory fusion data is processed using convolutional neural networks and graph attention networks to obtain global trajectory interaction data, which includes information on the influence of different pedestrians on pedestrian trajectories. Based on the global trajectory interaction data, the trajectories of each pedestrian in the current environment are predicted within a future preset time period.

2. The method according to claim 1, characterized in that, The determination of trajectory fusion data based on the pedestrian trajectory data and the scene data includes: Extract trajectory features from the pedestrian trajectory data and scene features from the scene data; A weighted scene feature vector is determined based on the trajectory features and the scene features. The weighted scene feature vector is used to indicate the degree of influence of different scenes on pedestrian trajectories. The pedestrian trajectory data and the weighted scene feature vector are concatenated to obtain the trajectory fusion data.

3. The method according to claim 2, characterized in that, The step of determining the weighted scene feature vector based on the trajectory features and the scene features includes: The trajectory features are then subjected to dimensionality-upgrading processing based on the first linear transformation layer to obtain the trajectory feature vector. The scene features are reduced in dimensionality using the second linear transformation layer to obtain the scene feature vector. The trajectory feature vector and the scene feature vector are concatenated to obtain a fused feature vector; The fused feature vector is processed using a multilayer perceptron and the Softmax activation function to obtain an attention weight vector; The scene features are processed based on the attention weight vector to obtain the weighted scene feature vector.

4. The method according to claim 1, characterized in that, The trajectory fusion data is processed using a convolutional neural network and a graph attention network to obtain global trajectory interaction data, including: The trajectory fusion data is processed based on the convolutional neural network to obtain local trajectory interaction data; The local interaction data of the trajectory is processed based on the graph attention network to obtain the global interaction data of the trajectory.

5. The method according to claim 4, characterized in that, The process of processing the trajectory fusion data based on the convolutional neural network to obtain trajectory local interaction data includes: Based on N The M-convolutional kernel performs feature capture on the trajectory fusion data; The feature data captured by the convolution kernel is nonlinearly activated based on the parameterized modified linear unit PReLU.

6. The method according to claim 4, characterized in that, The process of processing the local trajectory interaction data based on the graph attention network to obtain global trajectory interaction data includes: The nodes and spatiotemporal edges of the graph attention network are constructed based on the local interaction data of the trajectory. The dynamic attention weights between nodes are determined based on the nodes and the spatiotemporal edges. Based on the nodes and their corresponding dynamic attention weights in the graph attention network, the nodes are aggregated to obtain the global interaction data of the trajectory.

7. The method according to claim 1, characterized in that, The prediction of the trajectories of pedestrians in the current environment within a preset future time period based on the global trajectory interaction data includes: The spatial features of the trajectory global interaction data are extracted based on a multi-layer convolutional network decoder, and each convolutional layer of the multi-layer convolutional network decoder is connected to a parameterized corrected linear unit (PReLU). Based on the extracted spatial features, the trajectory probability distribution of each pedestrian within a preset time period is output. The trajectory probability distribution is used to indicate the probability value of the pedestrian moving in different directions. The direction with the highest probability value is determined as the pedestrian's target trajectory direction.

8. The method according to claim 3, characterized in that, The matrix format of the attention weight vector includes: (H W), the matrix format of the scene features includes: (H) W C), H, and W are used to determine the grid coordinates in the scene image. Each scene image contains H. W grids and C scene dimensions; The process of processing the scene features based on the attention weight vector to obtain the weighted scene feature vector includes: For any scene dimension, the value of each network is multiplied by its corresponding attention weight and summed to obtain the weighted scene feature vector. The matrix format of the weighted scene feature vector is as follows: 1 C.

9. A pedestrian trajectory prediction device based on scene information fusion, characterized in that, include: The acquisition module is used to acquire pedestrian trajectory data and scene data in the current environment; The determination module is used to determine trajectory fusion data based on the pedestrian trajectory data and the scene data, wherein the trajectory fusion data contains information on the impact of different scenes on pedestrian trajectories; The processing module is used to process the trajectory fusion data based on convolutional neural networks and graph attention networks to obtain global trajectory interaction data, which includes information on the influence of different pedestrians on pedestrian trajectories. The prediction module is used to predict the trajectory of each pedestrian in the current environment within a preset time period in the future, based on the global trajectory interaction data.

10. An electronic device, characterized in that, The device includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device performs the method according to any one of claims 1 to 8.

Images