A pedestrian trajectory prediction method based on rotation equivariant convolutional neural network

Abstract

The present invention discloses a pedestrian trajectory prediction method based on a rotationally equivariant convolutional neural network, belonging to the field of trajectory prediction technology. The method comprises the following steps: obtaining a ground RGB image and pedestrian historical trajectory data from a bird's-eye view and performing preprocessing to obtain a pedestrian trajectory dataset; inputting the ground RGB image in the pedestrian trajectory dataset into a semantic segmentation network for semantic segmentation to obtain a ground semantic segmentation map; obtaining a historical trajectory heat map based on the pedestrian historical trajectory data; obtaining encoding features based on the ground semantic segmentation map and the historical trajectory heat map; inputting the encoding features into a target point heat map decoder to output a target point heat map; performing feature fusion between the encoding features and the target point heat map, and inputting the fused map into a future trajectory prediction decoder to obtain a pedestrian trajectory prediction result. This method can achieve short-term and long-term multimodal future trajectory prediction of pedestrians from a bird's-eye view, and improves the model's prediction accuracy and robustness.

Description

Technical Field

[0001] The present invention belongs to the technical field of trajectory prediction, and in particular relates to a pedestrian trajectory prediction method based on a rotation equivariant convolutional neural network. Background Art

[0002] Trajectory prediction is the task of predicting the future location of pedestrians or other objects based on their past behavior and environmental cues. It is of great significance in fields such as autonomous driving, robot navigation, pedestrian tracking, and human-computer interaction. However, due to the complexity and uncertainty of pedestrian behavior, interactions, and scene dynamics, pedestrian trajectory prediction is a very challenging task compared to inferring physical laws based on regularly distributed data.

[0003] In recent years, with the development of deep learning, extracting features from sequential data and using them to predict the future has attracted research in many fields, such as weather forecasting, traffic prediction, and physical simulation. Compared to reasoning based on physical laws or specific rules, pedestrians have high mobility, are subject to fewer constraints, and have inherently multimodal characteristics, making pedestrian trajectory prediction a significant challenge.

[0004] Traditional work uses deterministic models to predict pedestrian trajectories, but these models cannot effectively capture this multimodal nature. This shortcoming becomes even more pronounced in long-term trajectory prediction. While recent work has demonstrated promising results using deep learning-based pedestrian trajectory prediction methods, current models still suffer from insufficient robustness. Summary of the Invention

[0005] To solve the above problems, the present invention provides the following solutions:

[0006] A pedestrian trajectory prediction method based on a rotation equivariant convolutional neural network includes the following steps:

[0007] Obtain ground RGB images and pedestrian historical trajectory data from a bird's-eye view;

[0008] Preprocessing the ground RGB image and pedestrian historical trajectory data to obtain a pedestrian trajectory dataset, and dividing the pedestrian trajectory dataset into a training set and a test set;

[0009] Input the ground RGB image in the training set into the semantic segmentation network for semantic segmentation to obtain the ground semantic segmentation map;

[0010] Obtain a historical trajectory heat map based on the pedestrian historical trajectory data in the training set;

[0011] Perform channel-dimensional splicing processing on the ground semantic segmentation map and the historical trajectory heat map to obtain a spliced map, input the spliced map into an encoder, encode features through a rotation equivariant convolution module, extract features, and learn rotation equivariance to obtain encoded features;

[0012] Input the image features into a target point heat map decoder and output a target point heat map;

[0013] The encoded features are fused with the target point heat map to obtain a feature fusion map, which is then input into a future trajectory prediction decoder to obtain a pedestrian trajectory prediction result.

[0014] Preferably, the method for obtaining the ground RGB image and pedestrian historical trajectory data from a bird's-eye view includes: shooting the ground RGB image and pedestrian historical trajectory data from a bird's-eye view by a drone; the ground RGB image includes pedestrians, roads, buildings, and vegetation; and the pedestrian historical trajectory data includes pedestrian coordinate position information.

[0015] Preferably, the preprocessing method includes: matching and aligning the pedestrian coordinate position information in the pedestrian historical trajectory data with the ground RGB image.

[0016] Preferably, the ground semantic segmentation map is obtained by a U-Net semantic segmentation network, and the U-Net semantic segmentation network segments the ground RGB image according to action categories, and the action categories include walking, standing, and running.

[0017] Preferably, the method for obtaining the historical trajectory heat map includes: establishing corresponding positional relationship coordinates through the pedestrian historical trajectory data and the spatial position of the ground background image taken by the drone; depicting the pedestrian position in the drone-taken image in an image of the same size as the ground RGB image taken by the drone to obtain the pedestrian historical trajectory map; and obtaining the historical trajectory heat map by heat map conversion.

[0018] Preferably, the neural network of the encoder includes three modules, wherein the first module includes a rotational equivariant convolution module, the second module includes a maximum pooling layer and two rotational equivariant convolution modules, and the third module includes a maximum pooling module.

[0019] Preferably, the rotational equivariant convolution module includes a rotational equivariant convolution layer and a ReLU activation function layer;

[0020] Based on the ReLU activation function, feature maps of the same size are obtained.

[0021] Preferably, the target point heat map decoder includes six modules, wherein the first module includes two rotated equilateral convolution modules, the second module includes an upsampling convolution module and two rotated equivariant convolution modules, the third module includes an upsampling convolution module and two rotated equivariant convolution modules, the fourth module includes an upsampling convolution module and two rotated equivariant convolution modules, the fifth module includes an upsampling convolution module and two rotated equivariant convolution modules, and the fifth module includes an upsampling convolution module and two rotated equivariant convolution modules.

[0022] Preferably, the upsampling convolution module includes an upsampling layer and a rotation equivariant convolution layer, and the spatial resolution of the information is increased by the upsampling convolution module;

[0023] The prediction module includes a rotational equivariant convolution layer and a sigmoid layer.

[0024] Preferably, the future trajectory prediction decoder and the target point heat map decoder have the same network structure.

[0025] Compared with the prior art, the present invention has the following advantages and technical effects:

[0026] The present invention predicts the future trajectory of pedestrians by designing a rotational equivariant convolutional neural network. The model training time is shortened, the prediction accuracy is higher, and it can achieve short-term and long-term multimodal future trajectory prediction of pedestrians from a bird's-eye view.

[0027] The present invention uses a rotational equivariant convolutional layer, which is different from the traditional classical convolutional layer. During the model training process, it can learn the equivariant characteristics of pedestrian motion and improve the model prediction accuracy and prediction robustness without adding additional parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0029] Figure 1 Schematic diagram of a pedestrian trajectory prediction method based on a rotation equivariant convolutional neural network according to an embodiment of the present invention;

[0030] Figure 2 This is a flow chart of a pedestrian trajectory prediction method based on a rotation equivariant convolutional neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0032] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Example 1

[0034] like Figure 1-2 As shown, the pedestrian trajectory prediction method based on the rotation equivariant convolutional neural network of this embodiment includes the following steps:

[0035] S1: Use drone photography to obtain RGB images of the ground background from a bird's-eye view and pedestrian historical trajectory data; the ground background RGB image includes scene information such as pedestrians, roads, buildings, and vegetation; the pedestrian historical trajectory data includes coordinate location information.

[0036] S2: Preprocess the data to obtain a pedestrian trajectory dataset, which is divided into a training set and a test set.

[0037] S3: Input the RGB image of the ground background from the bird's-eye view in the training set in step S2 into the semantic segmentation network for semantic segmentation to obtain a ground semantic segmentation map;

[0038] S4: Convert the observed historical pedestrian trajectories into a historical trajectory heatmap. First, establish the corresponding positional coordinates of the pedestrian's historical trajectory with the spatial position of the drone-captured ground background image corresponding to S2. Then, plot the pedestrian's position in eight consecutive camera frames over the past 3.2 seconds in an image of the same size as the drone-captured ground background image to obtain the pedestrian's historical trajectory map. Heatmap conversion is used to obtain the pedestrian's historical trajectory heatmap.

[0039] S5: Concatenate the ground semantic segmentation map obtained in step S3 and the historical trajectory heat map obtained in step S4 in the channel dimension and input them into the encoder E u In the encoder, the rotation equivariant convolution module is used to encode the features, extract the features and learn the rotation equivariance;

[0040] S6: The encoder E in step S5 is u The extracted features are input to the target point heat map decoder D g , output the target point heat map;

[0041] S7: The encoder E in step S5 is u The output of step S6 target point heat map decoder D g The output target point heat map is feature fused and input into the future trajectory prediction decoder D t, outputs the future trajectory distribution and predicts the pedestrian’s trajectory within 4.2 seconds;

[0042] The encoder E used in step S5 is u , the target point heat map decoder D used in step S6 g , the future trajectory prediction decoder D used in step S7 t , the network structure uses the rotation equivariant convolution layer (see Figure 1 In the upper middle part). Since most of the rotation angles in streets and intersections in road traffic scenes are 90°, the rotation equivariant operator selected in this embodiment belongs to the C4 cyclic group, specifically:

[0043]

[0044] Furthermore, in step S1,

[0045] The ground RGB images and pedestrian historical trajectory data from a bird's-eye view are obtained through drone photography and other means; the ground RGB images include scene information such as pedestrians, roads, buildings, vegetation, etc.; the pedestrian historical trajectory data includes coordinate location information.

[0046] Furthermore, in step S3,

[0047] The obtained ground RGB image is input into the U-Net semantic segmentation network to obtain the ground semantic segmentation map, which contains N C The system is composed of classes, which are determined by the action categories such as walking, standing, and running.

[0048] Furthermore, in step S4,

[0049] We adopt a trajectory representation on a scene heatmap to solve the alignment problem by spatially representing the trajectory in the same 2D space as the ground-truth image.

[0050] Furthermore, in step S5,

[0051] The encoder extracts features from the input information. The first module of the encoder's rotational equivariant convolution feature extraction part consists of a rotational equivariant convolution module, followed by four modules consisting of a maximum pooling layer and two rotational equivariant convolution modules, and the last module consists of a maximum pooling module. The above modules constitute the encoder's neural network.

[0052] The ground semantic segmentation map and the historical trajectory heat map are spliced and input into the encoder, which obtains the encoded features containing equivariant information. The encoder outputs the encoded features as the input of the target point heat map decoder and the future trajectory prediction decoder.

[0053] Preferably, in step S5,

[0054] The rotation equivariant convolution module is composed of a rotation equivariant convolution layer and a ReLU activation function layer; the rotation equivariant convolution layer encodes the hidden information and learns the rotation equivariance. Its convolution kernel size is 3×3, the step size is 1, and the padding is 1; the rotation equivariant convolution module needs to obtain feature maps of the same size through the ReLU activation function after performing the rotation equivariant convolution layer; the maximum pooling layer window size is 2×2, and the step size is 2;

[0055] Furthermore, in step S6,

[0056] The first module of the target point heat map decoder consists of two rotated equilateral convolution modules, and the following four modules are composed of one upsampling convolution module and two rotated equivariant convolution modules. In the first rotated equivariant convolution module of each decoding module, the obtained information is spliced with the corresponding layer information of the encoder; the last module of the decoder consists of one upsampling convolution module and one prediction module.

[0057] The target point heat map decoder inputs the encoded feature output of the encoder, and at the same time, the first rotational equivariant convolution module of each decoding module performs feature fusion with the information of the corresponding dimension in the encoder;

[0058] The upsampling convolution module is used to double the spatial resolution of information. It consists of one upsampling layer and one rotational equivariant convolution layer.

[0059] The prediction module outputs the target point distribution, which consists of 1 rotational equivariant convolution layer and 1 sigmoid layer;

[0060] The rotation equivariant convolution layer decodes the hidden information and learns rotation equivariance. Its convolution kernel size is 3×3, the stride is 1, and the padding is 1;

[0061] The rotation equivariant convolution module needs to obtain feature maps of the same size through the ReLU activation function after performing the rotation equivariant convolution layer;

[0062] Upsampling layer, whose convolution kernel size is 3×3, stride is 1, and padding is 1;

[0063] Furthermore, in step S7,

[0064] The future trajectory prediction decoder has the same network structure as the target point heat map decoder. Its input is the information obtained by the encoder. In each upsampling convolution module, the obtained information is spliced with the information of the corresponding layer in the target point heat map decoder and the encoder. The prediction module outputs the predicted future trajectory distribution of the pedestrian, and finally generates the future trajectory prediction result.

[0065] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A pedestrian trajectory prediction method based on rotation equivariant convolutional neural network, characterized in that: The following steps are involved: Obtain ground RGB images and pedestrian historical trajectory data from a bird's-eye view; Preprocessing the ground RGB image and pedestrian historical trajectory data to obtain a pedestrian trajectory dataset, and dividing the pedestrian trajectory dataset into a training set and a test set; Input the ground RGB image in the training set into the semantic segmentation network for semantic segmentation to obtain the ground semantic segmentation map; Obtain a historical trajectory heat map based on the pedestrian historical trajectory data in the training set; Perform channel-dimensional splicing processing on the ground semantic segmentation map and the historical trajectory heat map to obtain a spliced map, input the spliced map into an encoder, encode features through a rotation equivariant convolution module, extract features, and learn rotation equivariance to obtain encoded features; The encoder's neural network consists of three modules, the first of which includes a rotational equivariant convolution module, the second module includes a maximum pooling layer and two rotational equivariant convolution modules, and the third module includes a maximum pooling module; The rotation equivariant convolution module includes a rotation equivariant convolution layer and a ReLU activation function layer; Get feature maps of the same size based on the ReLU activation function; Input the image features into a target point heat map decoder and output a target point heat map; The target point heat map decoder includes six modules, wherein the first module includes two rotation equilateral convolution modules, the second module includes an upsampling convolution module and two rotation equivariant convolution modules, the third module includes an upsampling convolution module and two rotation equivariant convolution modules, the fourth module includes an upsampling convolution module and two rotation equivariant convolution modules, the fifth module includes an upsampling convolution module and two rotation equivariant convolution modules, and the fifth module includes an upsampling convolution module and two rotation equivariant convolution modules. The fifth module includes an upsampling convolution module and a prediction module; The encoded features are fused with the target point heat map to obtain a feature fusion map, which is then input into a future trajectory prediction decoder to obtain a pedestrian trajectory prediction result.

2. The pedestrian trajectory prediction method based on rotation equivariant convolutional neural network according to claim 1 is characterized in that: The method for obtaining a ground RGB image and pedestrian historical trajectory data from a bird's-eye view includes: using a drone to shoot a ground RGB image and pedestrian historical trajectory data from a bird's-eye view; the ground RGB image includes pedestrians, roads, buildings, and vegetation; and the pedestrian historical trajectory data includes pedestrian coordinate position information.

3. The pedestrian trajectory prediction method based on rotation equivariant convolutional neural network according to claim 1 is characterized in that: The preprocessing method includes: matching and aligning pedestrian coordinate position information in pedestrian historical trajectory data with a ground RGB image.

4. The pedestrian trajectory prediction method based on rotation equivariant convolutional neural network according to claim 1 is characterized in that: The ground semantic segmentation map is obtained by a U-Net semantic segmentation network, which segments the ground RGB image according to action categories, where the action categories include walking, standing, and running.

5. The pedestrian trajectory prediction method based on rotation equivariant convolutional neural network according to claim 1 is characterized in that: The method for obtaining a historical trajectory heat map includes: establishing corresponding positional relationship coordinates using pedestrian historical trajectory data and the spatial positions of a ground background image taken by a drone; depicting the pedestrian positions in the drone-taken image in an image of the same size as the drone-taken ground RGB image to obtain a pedestrian historical trajectory map; and obtaining the historical trajectory heat map by heat map conversion.

6. The pedestrian trajectory prediction method based on rotation equivariant convolutional neural network according to claim 1 is characterized in that: The upsampling convolution module includes an upsampling layer and a rotation equivariant convolution layer, and the spatial resolution of the information is increased by the upsampling convolution module; The prediction module includes a rotational equivariant convolution layer and a sigmoid layer.

7. The pedestrian trajectory prediction method based on rotation equivariant convolutional neural network according to claim 1 is characterized in that: The future trajectory prediction decoder has the same network structure as the target point heat map decoder.

Images