Pedestrian trajectory prediction method and device based on first-person perspective video

Abstract

The present disclosure provides a pedestrian trajectory prediction method based on a first-person perspective video, including: obtaining the observed trajectory features of the pedestrian; obtaining the pedestrian intention features; correcting the observed trajectory features to obtain the corrected observed trajectory features combined with the pedestrian intention; mapping the corrected observed trajectory features to the distribution space of the model that meets the first preset probability, sampling in the distribution space of the model that meets the first preset probability to obtain first sampling data, fusing the first sampling data with the corrected observed trajectory features to obtain a first sampling feature tensor; obtaining the predicted end point of at least one future predicted trajectory based on the first sampling feature tensor; generating at least one future predicted trajectory based on the predicted end point guidance based on the first sampling feature tensor and the predicted end point of at least one future predicted trajectory. The present disclosure also provides a pedestrian trajectory prediction device based on a first-person perspective video, an electronic device, a readable storage medium, and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and more particularly to a method, device, electronic device, readable storage medium and computer program product for predicting pedestrian trajectories based on first-person perspective video. Background Art

[0002] First-person pedestrian trajectory prediction is of great significance to the decision-making and control of autonomous vehicles. At present, autonomous vehicles are generally equipped with camera sensors to collect high-resolution first-person video image data around the vehicle. Pedestrians are important participants in urban traffic. They have complex behaviors and are vulnerable to harm. Accurately predicting the trajectory of pedestrians from the first-person perspective of the on-board camera can help vehicles take measures in advance to protect the lives and property of pedestrians and improve driving safety.

[0003] The perception systems in current autonomous vehicles generally integrate fairly mature and stable pedestrian detection and tracking algorithms, and the first-person pedestrian trajectory prediction task can be regarded as a downstream task of pedestrian detection and tracking.

[0004] In 2018, "Future person localization in first-person videos" proposed a framework for predicting the future position of pedestrians in first-person videos shot by wearable cameras. The wearer's self-motion will significantly affect the position of pedestrians in the video, and the scale of the target pedestrian is correlated with the first-person perspective principle. The posture of the pedestrian can represent its future movement. The framework combines four information: pedestrian scale, pedestrian position, pedestrian posture, and self-motion, achieving higher performance, but the pedestrian posture information cannot represent complex action semantics.

[0005] The 2019 paper "Forecasting pedestrian trajectory with machine-annotated training data" believes that pedestrian trajectory prediction based on the first-person perspective of an autonomous vehicle is important and challenging, but it is limited by the annotation of the data set. Therefore, a scalable machine annotation solution is introduced to solve the problem of lack of training data, using an automatic pedestrian detection and tracking algorithm to learn from unlabeled data to generate bounding boxes without manual annotation. In addition, a dynamic trajectory predictor (DTP) is proposed, which predicts pedestrian trajectories within one second in the future based on the preliminary trajectories of the uniform linear motion model (CV) and the uniformly accelerated linear motion model (CA), combined with the correction of the optical flow features of continuous frames, but the optical flow calculation is time-consuming and not conducive to real-time prediction.

[0006] The 2021 paper "BiTraP: Bi-Directional Pedestrian Trajectory Prediction with Multi-Modal Goal Estimation" uses a gated recurrent unit (GRU) to encode the observed trajectory of pedestrians under the framework of a conditional variational autoencoder, and designs a bidirectional GRU decoder based on the endpoint to reduce the cumulative error and achieve a high prediction accuracy in the first-person and third-person perspectives. However, the introduction of recurrent neural networks makes it impossible to parallelize training and does not have a global receptive field. In addition, only inputting single-modal information of the trajectory cannot represent the complex movement of pedestrians.

[0007] In summary, the prior art has the following technical problems:

[0008] First, it is impossible to efficiently extract the features of pedestrian observation trajectories. Pedestrian trajectory data has the characteristics of time series and one dimension. In existing studies, recurrent neural networks and attention mechanisms are mostly used for feature extraction. However, recurrent neural network units need to process pedestrian trajectories time by time, which cannot be parallelized, is inefficient during training, and does not have a global receptive field. The Transformer model based on the self-attention mechanism has a larger network capacity and global receptive field. It calculates an adaptive attention coefficient according to different inputs and can flexibly respond to trajectories in various situations. However, due to the low-dimensional characteristics of trajectory data, which are generally coordinate points in a plane coordinate system, the Transformer-based model is prone to overfitting of the trajectory feature extraction network, thereby affecting network performance. Therefore, the two network structures are not the best choice for extracting trajectory features.

[0009] Second, the observed trajectory features cannot characterize the complex behaviors of pedestrians. The first-person pedestrian trajectory is represented by coordinates. The observed trajectory sequence shows the pedestrian's historical movement trend, which helps predict its future trajectory. However, there is a possibility of sudden changes in the pedestrian's future movement direction and speed, such as sudden stops and sudden turns. Pedestrians also have complex behaviors, such as standing still and crossing the road. Therefore, relying entirely on the observed trajectory features will lead to trajectory prediction errors in a series of scenarios. Such complex scenarios can be regarded as difficult samples, which is a great challenge in the first-person trajectory prediction task. Summary of the invention

[0010] In order to solve at least one of the above technical problems, the present disclosure provides a pedestrian trajectory prediction method, device, electronic device, readable storage medium and computer program product based on first-person perspective video.

[0011] According to one aspect of the present disclosure, a method for predicting pedestrian trajectories based on a first-person perspective video is provided, comprising:

[0012] Obtaining the observation trajectory of pedestrians based on the first-person perspective video, obtaining the observation trajectory features of pedestrians based on the observation trajectory of pedestrians; obtaining the intention features of pedestrians based on the first-person perspective video;

[0013] Based on the observation trajectory feature and the pedestrian intention feature, a mixed feature is obtained, and based on the mixed feature, the observation trajectory feature is corrected to obtain a corrected observation trajectory feature combined with the pedestrian intention;

[0014] Mapping the corrected observation trajectory feature to a distribution space of a model that meets a first preset probability, sampling in the distribution space of the model that meets the first preset probability to obtain first sampling data, and fusing the first sampling data with the corrected observation trajectory feature to obtain a first sampling feature tensor;

[0015] Obtaining a predicted endpoint of at least one future predicted trajectory based on the first sampled feature tensor;

[0016] Based on the first sampling feature tensor and the predicted endpoint of the at least one future predicted trajectory, at least one future predicted trajectory guided by the predicted endpoint is generated.

[0017] According to at least one embodiment of the present disclosure, a pedestrian trajectory prediction method based on a first-person perspective video obtains a pedestrian's observation trajectory feature based on the pedestrian's observation trajectory, including:

[0018] The local and global features of the pedestrian's observation trajectory are extracted to obtain the pedestrian's observation trajectory features within a continuous period of time.

[0019] According to at least one embodiment of the present disclosure, a pedestrian trajectory prediction method based on a first-person perspective video is provided to extract local and global features of the pedestrian's observation trajectory, including:

[0020] Converting the data representation of each trajectory point of the observation trajectory from a low dimension to a high dimension through a linear mapping;

[0021] The converted high-dimensional trajectory points are passed through a local feature extractor to extract local features;

[0022] The local features are used to extract global features through a global feature extractor.

[0023] According to at least one embodiment of the present disclosure, a pedestrian trajectory prediction method based on a first-person perspective video is provided, wherein the pedestrian intention feature is obtained based on the first-person perspective video, including:

[0024] Extracting the spatiotemporal features of the video frame where the pedestrian is located in the observation trajectory through a video feature extraction network, and using the spatiotemporal features of the video frame as the pedestrian intention features of the pedestrian;

[0025] The spatiotemporal feature is a binary classification pedestrian intention feature indicating whether to take action, and the spatiotemporal feature is identified by a binary classifier of an action intention identifier to obtain a pedestrian intention type corresponding to the spatiotemporal feature.

[0026] According to at least one embodiment of the present disclosure, a pedestrian trajectory prediction method based on a first-person perspective video is provided, wherein the observed trajectory feature is modified to obtain a modified observed trajectory feature combined with the pedestrian's intention, including:

[0027] Concatenating the feature tensor representing the pedestrian's intention feature with the feature tensor representing the observation trajectory feature to obtain a mixed feature tensor;

[0028] Inputting the mixed feature tensor into a multilayer perceptron, and obtaining a control parameter representing a relationship between a feature tensor for improving the pedestrian intention feature and a feature tensor for the observed trajectory feature through learning by the multilayer perceptron;

[0029] The observation trajectory feature is modified based on the control parameter to obtain a modified observation trajectory feature combined with the pedestrian's intention.

[0030] According to the pedestrian trajectory prediction method based on first-person perspective video of at least one embodiment of the present disclosure, the distribution space of the model that meets the first preset probability is a multidimensional normal distribution space that obeys a preset mean and a preset variance.

[0031] According to the pedestrian trajectory prediction method based on first-person perspective video of at least one embodiment of the present disclosure, the distribution space of the model that meets the first preset probability is obtained based on the distribution space of the model that meets the second preset probability after training with the pedestrian's future true trajectory characteristics and the corrected observation trajectory characteristics combined with the pedestrian's intention.

[0032] According to at least one embodiment of the present disclosure, a pedestrian trajectory prediction method based on a first-person perspective video, obtaining a distribution space of a model that meets the first preset probability based on the distribution space of the model that meets the second preset probability, includes:

[0033] The pedestrian's future true trajectory features and the modified observed trajectory features combined with the pedestrian's intention are mapped into the distribution space of a model that meets the second preset probability. Through training, the distribution space of the model that meets the second preset probability is made to approach the distribution space of the model that meets the first preset probability.

[0034] According to at least one embodiment of the present disclosure, the method for predicting pedestrian trajectories based on first-person perspective video, training the model that meets the second preset probability includes:

[0035] Extract local and global features of pedestrians’ future true trajectory;

[0036] After splicing the local and global features of the pedestrian's future true trajectory with the features of the corrected observed trajectory combined with the pedestrian's intention, the features are input into a recognition network, where the recognition network is a model that meets the second preset probability;

[0037] Through training, the distribution space of the model that meets the second preset probability is made close to the distribution space of the model that meets the first preset probability.

[0038] According to at least one embodiment of the present disclosure, a pedestrian trajectory prediction method based on a first-person perspective video generates at least one future predicted trajectory guided by the predicted endpoint based on the first sampled feature tensor and the predicted endpoint of the at least one future predicted trajectory, including:

[0039] Based on the predicted end point of the future predicted trajectory, generate the intermediate process trajectory points of the future predicted trajectory;

[0040] Based on the intermediate process trajectory points of the future predicted trajectory and the predicted endpoint, a future predicted trajectory guided by the predicted endpoint is generated.

[0041] According to at least one embodiment of the present disclosure, a pedestrian trajectory prediction method based on a first-person perspective video generates intermediate process trajectory points of the future predicted trajectory based on the predicted end point of the future predicted trajectory, including:

[0042] Performing dimension upgrade on the predicted endpoint feature tensor based on linear mapping to obtain the predicted endpoint feature tensor after dimension upgrade;

[0043] Concatenate the first sampling feature tensor with the dimension-upgraded predicted endpoint feature tensor to obtain a mixed trajectory feature;

[0044] The hybrid trajectory features are input into a multi-layer perceptron to obtain intermediate process trajectory points of the future predicted trajectory guided by the predicted endpoint.

[0045] According to another aspect of the present disclosure, a pedestrian trajectory prediction device based on a first-person perspective video is provided, comprising:

[0046] A trajectory feature acquisition module, wherein the trajectory feature acquisition module acquires the pedestrian's observation trajectory based on the first-person perspective video, and acquires the pedestrian's observation trajectory features based on the pedestrian's observation trajectory;

[0047] A pedestrian intention recognition module, wherein the pedestrian intention recognition module obtains pedestrian intention features based on a first-person perspective video;

[0048] A trajectory correction module, wherein the trajectory correction module obtains a mixed feature based on the observed trajectory feature and the pedestrian intention feature, and corrects the observed trajectory feature based on the mixed feature to obtain a corrected observed trajectory feature combined with the pedestrian intention;

[0049] A priori network module, wherein the priori network module is a model that conforms to a first preset probability;

[0050] A sampling feature generation module, wherein the sampling feature generation module samples the corrected observation trajectory feature mapped to the distribution space of the model that meets the first preset probability to obtain first sampling data, and fuses the first sampling data with the corrected observation trajectory feature to obtain a first sampling feature tensor;

[0051] A trajectory endpoint generation module, wherein the trajectory endpoint generation module obtains a predicted endpoint of at least one future predicted trajectory based on the first sampling feature tensor;

[0052] A prediction trajectory generation module, wherein the prediction trajectory generation module generates at least one future prediction trajectory guided by the prediction endpoint based on the first sampling feature tensor and the prediction endpoint of the at least one future prediction trajectory.

[0053] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0054] A memory storing execution instructions;

[0055] A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the pedestrian trajectory prediction method of any embodiment of the present disclosure.

[0056] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the pedestrian trajectory prediction method of any embodiment of the present disclosure is used to implement.

[0057] According to another aspect of the present disclosure, a computer program product is provided, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, the pedestrian trajectory prediction method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0059] Figure 1It is a flowchart of a pedestrian trajectory prediction method based on first-person video according to an embodiment of the present disclosure.

[0060] Figure 2 It is a flowchart of a pedestrian trajectory prediction method based on first-person perspective video according to another embodiment of the present invention.

[0061] Figure 3 It is a flowchart of a method for mapping a motion trajectory combined with a pedestrian's motion intention into a distribution space that conforms to a first preset probability model or a second preset probability model according to an embodiment of the present disclosure.

[0062] Figure 4 It is a schematic diagram of an intention-based trajectory control and improvement method according to an embodiment of the present disclosure.

[0063] Figure 5 It is a structural diagram of a pedestrian trajectory prediction network based on first-person perspective video according to an embodiment of the present disclosure.

[0064] Figure 6 It is a schematic block diagram of the structure of a pedestrian trajectory prediction device based on first-person perspective video according to an embodiment of the present disclosure.

[0065] Figure 7 It is a schematic block diagram of the structure of a pedestrian trajectory prediction device based on first-person perspective video according to another embodiment of the present invention.

[0066] Description of Reference Numerals

[0067] 1000 Pedestrian Trajectory Prediction Device

[0068] 1002 Trajectory feature acquisition module

[0069] 1004 Pedestrian Intention Recognition Module

[0070] 1006 Trajectory Correction Module

[0071] 1008 Prior Network Module

[0072] 1010 Identify network module

[0073] 1012 Sampling feature generation module

[0074] 1014 Trajectory endpoint generation module

[0075] 1016 Prediction trajectory generation module

[0076] 1100 Bus

[0077] 1200 processor

[0078] 1300 Memory

[0079] 1400 Other circuits. DETAILED DESCRIPTION

[0080] The present disclosure is further described in detail below in conjunction with the accompanying drawings and implementations. It is understood that the specific implementations described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0081] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0082] Unless otherwise specified, the exemplary embodiments / implementations shown will be understood as providing exemplary features of various details of some ways in which the technical concept of the present disclosure can be implemented in practice. Therefore, unless otherwise specified, the features of the various embodiments / implementations can be combined, separated, interchanged and / or rearranged in addition without departing from the technical concept of the present disclosure.

[0083] The use of cross-hatching and / or shading in the accompanying drawings is generally used to make the boundaries between adjacent components clear. As such, unless otherwise specified, the presence or absence of cross-hatching or shading does not convey or indicate any preference or requirement for the specific materials, material properties, dimensions, proportions, commonalities between the components shown, and / or any other characteristics, attributes, properties, etc. of the components. In addition, in the accompanying drawings, the sizes and relative sizes of the components may be exaggerated for clarity and / or descriptive purposes. When the exemplary embodiments can be implemented differently, the specific process sequence can be performed in a different order than described. For example, two successively described processes can be performed substantially simultaneously or in an order opposite to the described order. In addition, the same figure numbers represent the same components.

[0084] When a component is referred to as being "on" or "over," "connected to," or "coupled to" another component, the component may be directly on, directly connected to, or directly coupled to the other component, or intervening components may be present. However, when a component is referred to as being "directly on," "directly connected to," or "directly coupled to" another component, there are no intervening components. For this purpose, the term "connected" may refer to a physical connection, an electrical connection, etc., with or without intervening components.

[0085] The terms used in the present disclosure are for the purpose of describing specific embodiments, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "one (kind, person)" and "said (the)" are also intended to include plural forms. In addition, when the terms "comprise" and / or "include" and their variations are used in this specification, it is explained that there are stated features, integral bodies, steps, operations, parts, assemblies and / or their groups, but one or more other features, integral bodies, steps, operations, parts, assemblies and / or their groups are not excluded from existence or addition. It should also be noted that, as used herein, the terms "substantially", "approximately" and other similar terms are used as approximate terms and not as degree terms, so that they are used to explain the inherent deviations of the measured values, calculated values ​​and / or the values ​​provided that will be recognized by those of ordinary skill in the art.

[0086] Combined with the following Figures 1 to 7 The pedestrian trajectory prediction method, device, electronic device, readable storage medium and computer program product based on first-person perspective video of the present invention are described in detail.

[0087] Figure 1 It is a flowchart of a pedestrian trajectory prediction method based on first-person perspective video according to an embodiment of the present disclosure.

[0088] refer to Figure 1 , a pedestrian trajectory prediction method S100 based on a first-person perspective video according to an embodiment of the present disclosure includes:

[0089] Obtaining the observation trajectory of pedestrians based on the first-person perspective video, obtaining the observation trajectory features of pedestrians based on the observation trajectory of pedestrians; obtaining the intention features of pedestrians based on the first-person perspective video;

[0090] Based on the observation trajectory features and the pedestrian intention features, a mixed feature is obtained, and the observation trajectory features are corrected based on the mixed feature to obtain a corrected observation trajectory feature combined with the pedestrian intention;

[0091] Mapping the corrected observation trajectory feature to the distribution space of the model that meets the first preset probability, sampling in the distribution space of the model that meets the first preset probability to obtain first sampling data, and fusing the first sampling data with the corrected observation trajectory feature to obtain a first sampling feature tensor;

[0092] Obtaining a predicted endpoint of at least one future predicted trajectory based on the first sampled feature tensor;

[0093] Based on the first sampled feature tensor and the predicted endpoint of at least one future predicted trajectory, at least one future predicted trajectory guided by the predicted endpoint is generated.

[0094] Figure 2It is a flowchart of a pedestrian trajectory prediction method based on first-person perspective video according to another embodiment of the present disclosure.

[0095] like Figure 2 As shown, the pedestrian trajectory prediction method S100 based on the first-person perspective video of the present disclosure may include a training phase and a prediction phase.

[0096] In the training phase, the observed trajectories, future true value trajectories and action intentions of pedestrians in several video frames in the video are trained so as to predict the future action trajectories of pedestrians in several subsequent video frames of the video frame of the observed trajectory. Specifically, it includes step S102 and step S104.

[0097] In step S102, the observation trajectory combined with the pedestrian's action intention is mapped into a distribution space that conforms to a first preset probability model, wherein the observation trajectory combined with the pedestrian's action intention is generated by correcting the observation trajectory by the pedestrian's action intention.

[0098] The video in the application scenario of the pedestrian trajectory prediction method disclosed in the present invention includes a first-person perspective video shot from the perspective of a moving vehicle during the autonomous driving process. For example, the autonomous driving vehicle is equipped with an optical camera to collect images of the road and scene in front of the vehicle in real time, and there are several pedestrians in the scene (it can be one pedestrian or more than two pedestrians). The detection module in the vehicle's autonomous driving perception system can detect the external frame of the pedestrian in the video image, and the tracking module in the autonomous driving perception system can associate the same pedestrian in different frames of the video. The first-person perspective video data in the present invention is preferably first pre-processed by the detection module and the tracking module, and then the first-person trajectory of the pedestrian in each frame is predicted in real time by the pedestrian trajectory prediction method disclosed in the present invention.

[0099] In the present disclosure, the observation time can be 0.5 seconds, the prediction time can be 1.5 seconds, the observation time corresponds to the observation trajectory, the prediction time corresponds to the future true value trajectory during training, and the prediction time corresponds to the future predicted trajectory during prediction.

[0100] In the present disclosure, for the motion trajectory of pedestrians, including the observed trajectory, the future true value trajectory, the observed trajectory combined with the pedestrian's action intention, and the future predicted trajectory involved in the following steps, it is preferred to use a unified trajectory and the representation of the trajectory points in the trajectory. Each trajectory point is generated based on the preprocessing of the first-person perspective video. For the original pedestrian external frame coordinates (x lt ,y lt ,x rb ,y rb), the present disclosure preferably converts the coordinates of the upper left corner and the lower right corner of the pedestrian's bounding box into the center point coordinates, width, height and their first-order differences. According to the principle of normalization, the processed coordinates are divided by the resolution of the video, and the coordinate values ​​are mapped between 0 and 1. In different data sets, the horizontal axis coordinates and the vertical axis coordinates are divided by the corresponding resolution values. More specifically, the observed trajectory of pedestrian i can be expressed as T represents the current moment, and the future true value trajectory of pedestrian i is expressed as Among them, the observation trajectory points are expressed as When the future true value trajectory point is used as input in the training phase, it is expressed as When calculating the error, no preprocessing is required, so it is expressed as Indicates the coordinate value of the center point and the width and height of the pedestrian's bounding box. represents the first-order difference of the center coordinate value and the width and height value of the pedestrian's external bounding box, m is the number of observation frames, n is the number of prediction frames, in the present disclosure, preferably, m is 15, n is 45, and the calculation method of related variables is as follows:

[0101]

[0102]

[0103] Among them, the coordinates of the corresponding points of the four corners of the pedestrian's external frame on the X-axis and Y-axis, Xrb represents the X-axis coordinate value of the corresponding point of the lower right corner of the external frame, Xlt represents the X-axis coordinate value of the corresponding point of the upper left corner of the external frame, Yrb represents the Y-axis coordinate value of the corresponding point of the lower right corner of the external frame, and Ylt represents the Y-axis coordinate value of the corresponding point of the upper left corner of the external frame.

[0104] In step S104, the pedestrian's future true value trajectory and the observed trajectory combined with the pedestrian's action intention are mapped to a distribution space that conforms to the second preset probability model. Through training, the distribution space of the second preset probability model is made close to the distribution space of the first preset probability model in step S102. After steps S102 and S104, the pedestrian trajectory training process of the first-person perspective video is completed.

[0105] In the prediction stage, the future movement trajectory of the pedestrian is predicted based on the observed trajectory and action intention of the pedestrian. After training, the trajectory of the pedestrian in the first-person perspective video can be predicted. The prediction process includes steps S106 to S112.

[0106] Because the observation trajectory needs to be encoded to generate observation trajectory features during both the training process and the prediction process. During the training process, it is also necessary to encode the observation trajectory in combination with the pedestrian's future true value trajectory to generate observation trajectory features (map the pedestrian's future true value trajectory and the observation trajectory combined with the pedestrian's action intention into a distribution space that conforms to the preset second preset probability model). Based on the relevant data information of the future true value trajectory, the distribution space that conforms to the first prediction probability model is adjusted, so that the distribution space that conforms to the second preset probability model approximates the distribution space that conforms to the first preset probability model, so as to perform future trajectory prediction. Therefore, Figure 3 A schematic flow chart of a method for mapping a motion trajectory combined with a pedestrian's motion intention into a distribution space that conforms to a first preset probability model or a preset second preset probability model (ie, a method for encoding pedestrian trajectories based on intention) is shown, such as Figure 3 As shown, the method S200 of mapping the action trajectory combined with the pedestrian's action intention into a distribution space that conforms to the first preset probability model or the preset second preset probability model includes the following steps.

[0107] In step S201, local and global features of the pedestrian observation trajectory are extracted. Specifically, for the observation trajectory X traj , first perform a linear mapping to enhance the original 8-dimensional input to 32 dimensions:

[0108]

[0109] in, Represents the linear layer and ReLU activation function. Then, local feature extraction of the trajectory is performed:

[0110]

[0111] in, The local feature extractor is preferably implemented by 1D convolution, with a convolution kernel size of 5. In order to keep the dimension of the feature tensor unchanged, the padding size is set to 2 and the number of 1D convolution layers is set to 4. The local feature extractor extracts trajectory points of 5 consecutive frames. Compared with the original trajectory sequence, the local features contain higher-level semantic information of the trajectory, which is beneficial to the subsequent feature extraction process and reduces the difficulty of training. Then, the local features of the trajectory are extracted. Input to the global feature extractor for global feature extraction:

[0112]

[0113] in, It is a global feature extractor, and this paper uses a multi-layer perceptron to implement it. First, the local features of the trajectory It is changed to 1-dimensional size, and then processed by a multi-layer perceptron and encoded into a 256-dimensional feature tensor. The global feature extractor encodes the complete pedestrian trajectory into a fixed feature tensor, extracting the trajectory features of the pedestrian in a continuous period of time, including the direction of movement, speed, etc. The multi-layer perceptron has a global receptive field for processing local features. The local trajectories at different times and channels are fully connected, fully exploring the inherent connections contained in the trajectory sequence coordinates.

[0114] In step S202, the pedestrian intention features based on the video are extracted. Specifically, a general video feature extraction network, such as TCN, TSM, TEA, etc., can be selected to extract the spatiotemporal features of the current video frame of the pedestrian, and pre-trained based on the pedestrian crossing intention recognition dataset, so that the network can extract potential factors that can affect the pedestrian trajectory, such as the pedestrian's delicate movement features, such as gait, body movement, head turn, etc., and finally, the pedestrian crossing intention is Figure 2 Feature tensor C before classification multi As the pedestrian intention feature, the feature tensor C multi After classification by the binary classifier, the obtained categories correspond to whether each action posture has the intention to cross the street. The video feature extraction network of this embodiment extracts the temporal and spatial features of pedestrians, and pre-trains the video feature extraction network based on the data set of pedestrian crossing intention recognition to identify pedestrian crossing intention. Figure 2 The feature tensor before classification is used as the pedestrian intention feature. The observed trajectory features are controlled and improved according to the intention features to achieve a more robust feature extraction effect.

[0115] In step S203, based on the pedestrian's intention features, the pedestrian's observation trajectory features are improved and controlled. Figure 4 As shown in the figure, a schematic diagram of the trajectory control and improvement method based on pedestrian action intention. Specifically, the pedestrian intention feature contains richer details and has semantics that cannot be represented by a single trajectory point coordinate. Therefore, the observation trajectory feature tensor can be controlled and improved through the pedestrian intention feature. Achieve a robust coding effect. Figure 2 Pedestrian intention feature tensor C before the classification layer multi , C multi With the observed trajectory X traj Encoded feature tensor Perform concatenation operation to obtain mixed feature tensors Learning C using multi-layer perceptron multi and The control and improvement relationship of is used to enhance the rationality of pedestrian intention characteristics, as shown in the following formula:

[0116]

[0117] Among them, the output C of the multi-layer perceptron multi,X The dimensions and Similarly, after the Sigmoid activation function, C multi,X Mapped to the range [0,1], as shown below:

[0118] β=Sigmoid(C multi,X )

[0119] Finally, the control variable β (i.e., β as a control parameter for improving the relationship between the feature tensor of the pedestrian intention feature and the feature tensor of the observation trajectory feature) and the feature tensor encoded by the observation trajectory Perform Hadamard product, and use the control variable β as a mask to change the original In order to achieve the effect of controlling and improving trajectory characteristics, as shown in the following formula:

[0120]

[0121] In step S204, the pedestrian's observed trajectory features are mapped to a mean value of The variance is Specifically, the prior network P(Z|X,C) fits the probability distribution of the latent space tensor Z based on the pedestrian’s trajectory information and intention information, and the condition X,C is equivalent to the modified trajectory feature In this disclosure, given a strong assumption, the prior network P(Z|X,C) generates a mean The variance is Since the mean and variance cannot be directly calculated by the modified trajectory features, a neural network model is used instead:

[0122]

[0123] First, the corrected trajectory features Input into the multi-layer perceptron, output 64-dimensional feature tensor, divided into two parts, namely the 32-dimensional mean and the 32-dimensional variance

[0124] In step S205, it is determined whether it is in the training stage. If it is not in the training stage (ie, in the inference stage), then jump to step S206; if it is in the training stage, then jump to step S207.

[0125] In step S206, the mean and variance of the normal distribution and the observed trajectory characteristics of the pedestrian are output. Specifically, the mean of the multidimensional normal distribution (i.e., the second preset probability model) mapped by the prior network P(Z|X,C) is output. variance And the observed trajectory characteristics

[0126] In step S207, local and global features of the pedestrian's future true value trajectory are extracted. Specifically, for the pedestrian's future true value trajectory Y traj , first perform a linear mapping to enhance the original 8-dimensional input to 32 dimensions:

[0127]

[0128] in, Represents the linear layer and ReLU activation function. Then, local feature extraction of the trajectory is performed:

[0129]

[0130] in, The local feature extractor is implemented by 1D convolution, with a convolution kernel size of 5. In order to keep the dimension of the feature tensor unchanged, the padding size is set to 2 and the number of 1D convolution layers is set to 4. The local feature extractor extracts the trajectory points of 5 consecutive frames and abstracts the local features. Input to the global feature extractor for global feature extraction:

[0131]

[0132] in, It is a global feature extractor, and this paper uses a multi-layer perceptron to implement it. First, the local features of the trajectory It is changed to a 1-dimensional size, and then processed by a multi-layer perceptron to encode the complete pedestrian's future trajectory into a 256-dimensional feature tensor.

[0133] In step S208, the observed trajectory features of the pedestrian are mapped to a feature map having a mean value of μ Zq , the variance is The recognition network Q(Z|X,Y,C) fits the probability distribution of the latent space tensor Z based on the pedestrian's future trajectory truth information, observed trajectory information, and intention information. Among them, the future trajectory truth value Y traj After being encoded by local and global feature extractors, The observed trajectory information and intention information are converted into modified trajectory features Will and Perform concatenation operation to obtain As the input of the recognition network. Similar to the prior network, in this disclosure, the recognition network is strongly assumed to obey the mean μ Zq , the variance is The normal distribution of , using a neural network model instead of calculation:

[0134]

[0135] Specifically, Input into the multi-layer perceptron, output 64-dimensional feature tensor, divided into two parts, namely the 32-dimensional mean and the 32-dimensional variance

[0136] In step S209, the KLD loss function is used to make the normal distribution P approximate the normal distribution Q. Specifically, in order to make the prior network P(Z|X,C) represent the correct probability distribution, P(Z|X,C) is approximated to the recognition network Q(Z|X,Y,C) and trained using the KLD loss function, as shown in the following formula:

[0137] Loss KLD =D KL (Q(Z|X,Y,C)||P(Z|X,C))

[0138] In step S210, the mean and variance of the normal distribution and the observed trajectory features of the pedestrian are output. Specifically, the mean μ of the normal distribution Q(Z|X,Y,C) mapped by the prior network is output. Zq ,variance And the observed trajectory characteristics

[0139] In step S106, the observed trajectory combined with the pedestrian's action intention is mapped into a distribution space that conforms to a first preset probability model, and sampling is performed in the distribution space of the first preset probability model to obtain first sampling data.

[0140] In step S108, the first sampling data is concatenated with the observation trajectory features combined with the pedestrian intention features to form a first sampling feature tensor.

[0141] In the specific implementation of step S106 and step S108, sampling is performed in the latent space (the distribution space of the first preset probability model) and the trajectory features of the pedestrian are integrated. Specifically, in the test phase (i.e., the prediction phase), the latent space tensor Z of the prior network is directly p Sample K times to obtain K sample variables Will With the corrected trajectory characteristics Concatenate (Concat) to obtain the sampled feature tensor Correspondingly, during the training phase, by identifying the latent space tensor Z of the network q Sample K times to obtain K sample variables Will and corrected observation trajectory characteristics Perform concatenation operation, that is, the obtained sampling feature tensor is

[0142] In step S110, the first sampled feature tensor is input into the trajectory endpoint generator to generate a predicted endpoint in the future predicted trajectory, and the future predicted trajectory includes multiple predicted endpoints. According to the sampled feature tensor, the endpoints of the pedestrian's future multiple trajectories are generated. Specifically, first, for a given sampled feature tensor The end point of the future trajectory is generated by the trajectory end point generator. In this disclosure, the trajectory end point generator is implemented by a multi-layer perceptron:

[0143]

[0144] Among them, the output dimension of the multi-layer perceptron is 4, which represents the center point coordinates, width and height of the rectangular outer frame of the future trajectory endpoint, expressed as In the framework of conditional variational autoencoder, the latent space is sampled K times, and K endpoints are obtained after passing through the trajectory endpoint generator. Since the endpoint of the generated trajectory directly determines the center point displacement error CFDE and significantly affects the generation of the process trajectory, in order to improve the accuracy of the network in generating the endpoint and ensure that the generated endpoints are diverse rather than convergent, the optimal endpoint loss is added to the loss function, as shown below:

[0145]

[0146] Among them, under the loose strategy, the closest to the true value among the K endpoints is selected One to calculate Loss endpont .

[0147] In step S112, based on each predicted endpoint, intermediate process trajectory points between each predicted endpoint are generated, and the intermediate process trajectory points and the predicted endpoint together constitute the future predicted trajectory. Based on the predicted endpoint guidance, the pedestrian's future intermediate process trajectory is generated. Specifically, based on the sampled feature tensor and the predicted endpoint, the process trajectory is generated in a fully connected manner, the cumulative error is minimized, the parallelization of future trajectory generation is achieved, the correlation between the generated process trajectories is enhanced, and each process trajectory point has a global receptive field.

[0148] First, the end point coordinates After linear mapping, the dimension enhancement is performed as shown in the following formula:

[0149]

[0150] Sampling feature tensors contains the modified trajectory features and latent space tensor, With enhanced endpoint features Perform concatenation operation to obtain mixed trajectory features As shown below:

[0151]

[0152] Then, the mixed trajectory features Input into the multi-layer perceptron, directly decode the future trajectory based on the end point guidance, that is, the intermediate process trajectory, as shown in the following formula:

[0153]

[0154] In order to constrain the predicted intermediate process trajectory Set the loss function of the process trajectory, adopt a loose strategy, and only select the process trajectory that matches the true value The loss value with the smallest gap is returned by gradient, as shown in the following formula:

[0155]

[0156] It should be noted that, in each of the above steps S102 to S112, the observed trajectory and the future true trajectory are the motion trajectories of the pedestrians extracted from the video frames containing the pedestrian's motion, and the video frames corresponding to the future true trajectory are after the sequence of video frames corresponding to the observed trajectory. The observed trajectory, the future true trajectory or the observed trajectory combined with the pedestrian's action intention is represented by a combination of multiple trajectory points in each trajectory, and each trajectory point in the trajectory is represented by the size of the circumscribed rectangle based on the position of the pedestrian in the video frame and covering the pedestrian's boundary.

[0157] Figure 5 It is a structural diagram of a first-person perspective pedestrian trajectory prediction network based on intent according to an embodiment of the present disclosure.

[0158] like Figure 5 As shown in the figure, the first-person perspective pedestrian trajectory prediction network structure based on intention includes:

[0159] The multimodal pedestrian intention recognition network receives multimodal information as input and extracts video-based pedestrian intention features as output, where pedestrian action intentions include multiple types. Specifically, multimodal information is represented by a video frame sequence, and multimodal information can also include vehicle speed, pedestrian trajectory, posture, etc. Specifically, the recognition network can use a general video feature extraction network, such as TCN, TSM, TEA, etc., by extracting the spatiotemporal features of the current video frame of the pedestrian, and pre-training based on the pedestrian crossing intention recognition dataset, so that the pedestrian intention network can extract potential factors that can affect the pedestrian trajectory, such as the pedestrian's delicate movement features, such as gait, body movement, head turning, etc., and then integrate other modal features. Finally, the pedestrian's crossing intention is Figure 2 Feature tensor C before classification multi As pedestrian intention features.

[0160] The local and global trajectory feature extractor receives the observed trajectory X traj Or the future truth trajectory Y traj The local and global trajectory feature extractors of this embodiment first extract features of the local trajectory of pedestrians at a low computational cost to capture the trajectory attributes of pedestrians in a short time interval, and then use the global feature extractor to encode the complete pedestrian features under the full receptive field to obtain the trajectory feature tensor.

[0161] The trajectory control and improvement module receives data representing the pedestrian's action intention and data representing the observed trajectory, corrects the observed trajectory based on the pedestrian's intention characteristics and outputs the motion trajectory combined with the pedestrian's action intention.

[0162] The prior network module receives the observed trajectory of the pedestrian combined with the pedestrian intention characteristics output by the trajectory control and improvement module, and maps the observed trajectory characteristics of the pedestrian into a mean The variance is In this embodiment, given a strong assumption, the prior network P(Z|X,C) generates a multidimensional normal distribution P with a mean of The variance is Since the mean and variance cannot be directly calculated by the modified trajectory features, a neural network model is used instead:

[0163]

[0164] First, the corrected trajectory features Input into the multi-layer perceptron, output 64-dimensional feature tensor, divided into two parts, namely the 32-dimensional mean and the 32-dimensional variance

[0165] Output the mean and variance of the normal distribution and the observed trajectory characteristics of the pedestrian. Specifically, output the mean of the multidimensional normal distribution mapped by the prior network P(Z|X,C) variance And the observed trajectory characteristics

[0166] The recognition network module maps the observed trajectory features of pedestrians to a mean value of μ Zq , the variance is The recognition network Q(Z|X,Y,C) fits the probability distribution of the latent space tensor Z based on the pedestrian's future trajectory truth information, observed trajectory information, and action intention information. Among them, the future trajectory truth value Y traj After being encoded by local and global feature extractors, The observed trajectory information and intention information are converted into modified trajectory features Will and Perform concatenation operation to obtain As the input of the recognition network. Similar to the prior network, in this disclosure, the recognition network is strongly assumed to obey the mean μ Zq , the variance is The normal distribution of , using a neural network model instead of calculation:

[0167]

[0168] Specifically, Input into the multi-layer perceptron, output 64-dimensional feature tensor, divided into two parts, namely the 32-dimensional mean and the 32-dimensional variance

[0169] At this time, the KLD loss function is used to make the normal distribution P approximate the normal distribution Q in order to complete the training, and then the trajectory prediction is performed through the already trained first-person perspective pedestrian trajectory prediction structure based on intention. Specifically, in order to make the prior network P(Z|X,C) represent the correct probability distribution, P(Z|X,C) is approximated to the recognition network Q(Z|X,Y,C), and the KLD loss function is used for training, as shown in the following formula:

[0170] Loss KLD =D KL (Q(Z|X,Y,C)||P(Z|X,C))

[0171] The trajectory endpoint generator samples in the data space generated by the prior network module, and based on the sampled data, generates the predicted endpoints in the future predicted trajectory, which includes multiple predicted endpoints. Specifically, in the test phase, directly in the hidden space tensor Z of the prior network p Sample K times to obtain K sample variables Will With the corrected trajectory characteristics Concatenate (Concat) to obtain the sampled feature tensor

[0172] According to the sampled feature tensor, the end points of the pedestrian's future multi-trajectory are generated. Specifically, first, for a given sampled feature tensor The end point of the future trajectory is generated by the trajectory end point generator. In the present disclosure, the trajectory end point generator can be implemented by a multi-layer perceptron:

[0173]

[0174] Preferably, the output dimension of the multilayer perceptron is 4, which respectively represent the center point coordinates and the width and height values ​​of the rectangular bounding box of the future trajectory endpoint, expressed as In the framework of conditional variational autoencoder, the latent space is sampled K times, and K endpoints are obtained after passing through the trajectory endpoint generator. Since the endpoint of the generated trajectory directly determines the center point displacement error CFDE and significantly affects the generation of the process trajectory, in order to improve the accuracy of the network in generating the endpoint and ensure that the generated endpoints are diverse rather than convergent, the optimal endpoint loss is added to the loss function, as shown below:

[0175]

[0176] Among them, under the loose strategy, the closest to the true value among the K endpoints is selected One to calculate Loss endpont .

[0177] The process trajectory generator generates intermediate process trajectory points between the predicted endpoints based on the predicted endpoints, and the intermediate process trajectory points and the predicted endpoints together constitute the future predicted trajectory. That is, based on the endpoint guidance, the pedestrian's future intermediate process trajectory is generated. Specifically, based on the sampled feature tensor and the predicted endpoint, the process trajectory is generated in a fully connected manner to minimize the cumulative error, realize parallel future trajectory generation, enhance the correlation between the generated process trajectories, and each process trajectory point has a global receptive field. First, the endpoint coordinates are After linear mapping, the dimension enhancement is performed as shown in the following formula:

[0178]

[0179] Sampling feature tensors contains the modified trajectory features and latent space tensor, With enhanced endpoint features Perform concatenation operation to obtain mixed trajectory features As shown below:

[0180]

[0181] Then, the mixed trajectory features Input into the multi-layer perceptron, directly decode the future trajectory based on the end point guidance, that is, the intermediate process trajectory, as shown in the following formula:

[0182]

[0183] In order to constrain the predicted intermediate process trajectory Set the loss function of the process trajectory, adopt a loose strategy, and only select the process trajectory that matches the true value The loss value with the smallest gap is returned by gradient, as shown in the following formula:

[0184]

[0185] It should be noted that the technical implementation details involved in the relevant modules in this embodiment correspond to the pedestrian trajectory prediction method based on the first-person perspective video disclosed in the present invention, and will not be repeated here.

[0186] The conditional variational autoencoder framework adopted in this embodiment is specifically a multi-trajectory prediction framework based on conditional variational autoencoder. The implementation idea of ​​the conditional variational autoencoder is to map the input to a normal distribution, and then sample the normal distribution to obtain multiple outputs, which just meets the application scenario requirements of "input one trajectory and output multiple possible trajectories". It should be noted that in this embodiment, based on the conditional variational autoencoder framework, replacing it with a generative adversarial network can also achieve multi-trajectory prediction.

[0187] The pedestrian trajectory prediction network structure based on the first-person perspective video provided in this embodiment has a network architecture divided into two stages: encoder and decoder. In the encoder stage, the input is mapped to the probability distribution of the latent space tensor, and the decoder enables the network to generate multiple future trajectories through multiple sampling in the latent space tensor. Since low-dimensional trajectory features cannot represent the complex behavior of pedestrians, and the intention features of pedestrians will significantly affect the trajectory of pedestrians, this paper improves and controls the observed trajectory features based on the pedestrian intention features extracted based on video data to improve the accuracy of trajectory prediction.

[0188] According to an embodiment of the present disclosure, a pedestrian trajectory prediction device 1000 based on a first-person perspective video includes:

[0189] A trajectory feature acquisition module 1002, the trajectory feature acquisition module 1002 acquires the pedestrian's observation trajectory based on the first-person perspective video, and acquires the pedestrian's observation trajectory features based on the pedestrian's observation trajectory;

[0190] A pedestrian intention recognition module 1004, which obtains pedestrian intention features based on the first-person perspective video;

[0191] The trajectory correction module 1006 obtains a mixed feature based on the observed trajectory feature and the pedestrian intention feature, and corrects the observed trajectory feature based on the mixed feature to obtain a corrected observed trajectory feature combined with the pedestrian intention;

[0192] A priori network module 1008, the priori network module 1008 is a model that meets the first preset probability;

[0193] A sampling feature generation module 1012, which samples the corrected observation trajectory features mapped to the distribution space of the model that meets the first preset probability to obtain first sampling data, and fuses the first sampling data with the corrected observation trajectory features to obtain a first sampling feature tensor;

[0194] A trajectory endpoint generation module 1014, the trajectory endpoint generation module 1014 obtains a predicted endpoint of at least one future predicted trajectory based on the first sampled feature tensor;

[0195] The predicted trajectory generation module 1016 generates at least one future predicted trajectory guided by the predicted endpoint based on the first sampled feature tensor and the predicted endpoint of at least one future predicted trajectory.

[0196] The pedestrian trajectory prediction device 1000 based on the first-person perspective video of the present disclosure can be completely implemented by means of a computer software program architecture. Figure 6 , can also be implemented based on the hardware architecture using a processor.

[0197] According to another embodiment of the present disclosure, a pedestrian trajectory prediction device 1000 based on a first-person perspective video includes:

[0198] The trajectory feature acquisition module 1002 extracts local features from the pedestrian's motion trajectory, and extracts global features based on the local features.

[0199] The pedestrian intention recognition module 1004 is configured to extract pedestrian intention features of pedestrians corresponding to the pedestrian motion trajectory from the video frames of pedestrian motion.

[0200] The trajectory correction module 1006 corrects the local and global features representing the pedestrian's motion trajectory in combination with the pedestrian's action intention to obtain a corrected pedestrian motion trajectory.

[0201] The prior network module 1008 maps the observed trajectory combined with the pedestrian's action intention into a distribution space that conforms to the first preset probability model.

[0202] The identification network module 1010 maps the pedestrian's future true value trajectory and the observed trajectory combined with the pedestrian's action intention into a distribution space that conforms to a preset second preset probability model. Through training, the distribution space of the second preset probability model approaches the distribution space of the first preset probability model.

[0203] The sampling feature generation module 1012 samples in the distribution space of the first preset probability model to obtain first sampling data, and concatenates the first sampling data with the observation trajectory features combined with the pedestrian intention features to form a first sampling feature tensor.

[0204] The trajectory endpoint generation module 1014 inputs the first sampling feature tensor into the trajectory endpoint generation module to generate a predicted endpoint in the future predicted trajectory, and the future predicted trajectory includes multiple predicted endpoints.

[0205] The prediction trajectory generation module 1016 generates intermediate process trajectory points between the prediction endpoints based on the prediction endpoints, and the intermediate process trajectory points and the prediction endpoints together constitute the future prediction trajectory.

[0206] The pedestrian trajectory prediction device based on the first-person perspective video of this embodiment can also be implemented by means of a computer software program architecture, or can be implemented as follows: Figure 7 As shown, the hardware architecture of the processor is adopted for implementation.

[0207] The method and device for predicting pedestrian trajectories from a first-person perspective video based on intentions provided by the present invention use a video feature extraction network to extract the temporal and spatial features of pedestrians in a high-resolution video taken by a vehicle-mounted camera, and pre-train the video feature extraction network based on a dataset of pedestrian crossing intention recognition to identify pedestrian crossing intentions. Figure 2 The feature tensor before classification is used as the pedestrian intention feature. Based on the framework of conditional variational autoencoder, multiple possible future trajectories of pedestrians are predicted according to the observed trajectory and intention features of pedestrians. In the encoder, local and global trajectory feature extractors are designed, and an intention-based trajectory control and improvement module is designed, which adjusts and controls the observed trajectory features through the potential intention of pedestrians to cope with more complex pedestrian motion semantics. In the decoder, the end point of the future trajectory is generated, and the process trajectory is decoded under the guidance of the end point to prevent error accumulation.

[0208] The pedestrian trajectory prediction device 1000 based on the first-person perspective video of the present disclosure may include corresponding modules for executing each or several steps in the above flowchart. Therefore, each step or several steps in the above flowchart can be executed by the corresponding module, and the device may include one or more modules of these modules. The module can be one or more hardware modules specially configured to execute the corresponding steps, or implemented by a processor configured to execute the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0209] The hardware structure can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0210] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0211] Any process or method description in the flowchart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present disclosure includes other implementations, in which the functions may not be performed in the order shown or discussed, including performing the functions in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the present disclosure. The processor performs the various methods and processes described above. For example, the method implementation in the present disclosure can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be performed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods in any other appropriate manner (e.g., by means of firmware).

[0212] The logic and / or steps represented in the flowchart or otherwise described herein may be embodied in any readable storage medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions on an instruction execution system, apparatus, or device), or in combination with such instruction execution systems, apparatuses, or devices.

[0213] For the purposes of this specification, a "readable storage medium" may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples of readable storage media (a non-exhaustive list) include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, the readable storage medium may even be paper or other suitable medium on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a memory.

[0214] It should be understood that the various parts of the present disclosure can be implemented in hardware, software, or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented in software stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0215] A person skilled in the art may understand that all or part of the steps of implementing the above-mentioned implementation method may be completed by instructing related hardware through a program, and the program may be stored in a readable storage medium, which, when executed, includes one or a combination of the steps of the implementation method.

[0216] In addition, each functional unit in each embodiment of the present disclosure may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a readable storage medium. The storage medium may be a read-only memory, a disk or an optical disk, etc.

[0217] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present application. In this specification, the schematic representations of the above terms are not necessarily the same embodiment / method or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments / methods or examples described in this specification and the features of the different embodiments / methods or examples, unless they are contradictory.

[0218] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0219] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A pedestrian trajectory prediction method based on first-person perspective video, characterized in that: include: Obtaining the observation trajectory of the pedestrian based on the first-person perspective video, and obtaining the observation trajectory features of the pedestrian based on the observation trajectory of the pedestrian; Obtain pedestrian intention features based on first-person perspective video; Based on the observation trajectory feature and the pedestrian intention feature, a mixed feature is obtained, and based on the mixed feature, the observation trajectory feature is corrected to obtain a corrected observation trajectory feature combined with the pedestrian intention; Mapping the corrected observation trajectory feature to a distribution space of a model that meets a first preset probability, sampling in the distribution space of the model that meets the first preset probability to obtain first sampling data, and fusing the first sampling data with the corrected observation trajectory feature to obtain a first sampling feature tensor; Obtaining a predicted endpoint of at least one future predicted trajectory based on the first sampled feature tensor; as well as At least one future prediction trajectory guided by the prediction endpoint is generated based on the first sampling feature tensor and the prediction endpoint of the at least one future prediction trajectory.

2. The pedestrian trajectory prediction method based on first-person perspective video according to claim 1, characterized in that: The observation trajectory features of the pedestrian are obtained based on the observation trajectory of the pedestrian, including: The local and global features of the pedestrian's observation trajectory are extracted to obtain the pedestrian's observation trajectory features within a continuous period of time.

3. The pedestrian trajectory prediction method based on first-person perspective video according to claim 2 is characterized in that: Extract local and global features of pedestrian observation trajectories, including: Converting the data representation of each trajectory point of the observation trajectory from a low dimension to a high dimension through a linear mapping; Extracting local features from the converted high-dimensional trajectory points through a local feature extractor; and The local features are used to extract global features through a global feature extractor.

4. The pedestrian trajectory prediction method based on first-person perspective video according to claim 2, characterized in that: Obtain pedestrian intention features based on first-person perspective video, including: Extracting the spatiotemporal features of the video frame where the pedestrian is located in the observation trajectory through a video feature extraction network, and using the spatiotemporal features of the video frame as the pedestrian intention features of the pedestrian; The spatiotemporal feature is a binary classification pedestrian intention feature indicating whether to take action, and the spatiotemporal feature is identified by a binary classifier of an action intention identifier to obtain a pedestrian intention type corresponding to the spatiotemporal feature.

5. The pedestrian trajectory prediction method based on first-person perspective video according to claim 2, characterized in that: The observation trajectory feature is modified to obtain a modified observation trajectory feature combined with the pedestrian's intention, including: Concatenating the feature tensor representing the pedestrian's intention feature with the feature tensor representing the observation trajectory feature to obtain a mixed feature tensor; Inputting the mixed feature tensor into a multilayer perceptron, and obtaining, through learning by the multilayer perceptron, a control parameter representing a relationship between a feature tensor for improving the pedestrian intention feature and a feature tensor for the observed trajectory feature; and The observation trajectory feature is modified based on the control parameter to obtain a modified observation trajectory feature combined with the pedestrian's intention.

6. The method for predicting pedestrian trajectories based on first-person perspective video according to claim 1, characterized in that: The distribution space of the model that meets the first preset probability is a multidimensional normal distribution space that obeys a preset mean and a preset variance.

7. A pedestrian trajectory prediction device based on first-person perspective video, characterized in that: include: A trajectory feature acquisition module, wherein the trajectory feature acquisition module acquires the pedestrian's observation trajectory based on the first-person perspective video, and acquires the pedestrian's observation trajectory features based on the pedestrian's observation trajectory; A pedestrian intention recognition module, wherein the pedestrian intention recognition module obtains pedestrian intention features based on a first-person perspective video; A trajectory correction module, wherein the trajectory correction module obtains a mixed feature based on the observed trajectory feature and the pedestrian intention feature, and corrects the observed trajectory feature based on the mixed feature to obtain a corrected observed trajectory feature combined with the pedestrian intention; A priori network module, wherein the priori network module is a model that conforms to a first preset probability; A sampling feature generation module, wherein the sampling feature generation module samples the corrected observation trajectory feature mapped to the distribution space of the model that meets the first preset probability to obtain first sampling data, and fuses the first sampling data with the corrected observation trajectory feature to obtain a first sampling feature tensor; a trajectory endpoint generation module, the trajectory endpoint generation module obtaining a predicted endpoint of at least one future predicted trajectory based on the first sampled feature tensor; and A prediction trajectory generation module, wherein the prediction trajectory generation module generates at least one future prediction trajectory guided by the prediction endpoint based on the first sampling feature tensor and the prediction endpoint of the at least one future prediction trajectory.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instruction stored in the memory, so that the processor executes the pedestrian trajectory prediction method according to any one of claims 1 to 6.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the pedestrian trajectory prediction method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the pedestrian trajectory prediction method described in any one of claims 1 to 6 is implemented.

Images