A multimodal pedestrian trajectory prediction method, system, device and medium

Abstract

The present invention discloses a multimodal pedestrian trajectory prediction method, system, device, and medium. The multimodal pedestrian trajectory prediction method comprises the following steps: obtaining an observed pedestrian trajectory to be predicted; and based on the obtained observed pedestrian trajectory, using a pre-trained multimodal pedestrian trajectory prediction model to perform future trajectory prediction, thereby obtaining a multimodal pedestrian trajectory prediction result. The technical solution disclosed in the present invention addresses the drawback of existing multimodal pedestrian trajectory prediction methods that ignore intermediate intentions. A multimodal pedestrian trajectory prediction method based on multiple intentions is specifically proposed, which can improve the accuracy and reliability of pedestrian trajectory prediction results.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology and relates to the field of autonomous driving, and in particular to a multimodal pedestrian trajectory prediction method, system, device and medium. Background Art

[0002] Pedestrian trajectory prediction involves generating predicted trajectories for a given set of observed pedestrians in a real-world scenario. This task has important applications in many real-world scenarios. For example, in autonomous driving, cars can use this method to determine pedestrians' future trajectories and make decisions in advance. In security, security cameras can use this method to track pedestrians' current and possible future locations.

[0003] Currently, existing pedestrian trajectory prediction is mainly based on a deep learning framework, which predicts future trajectories by learning the features of observed trajectories. However, it still has the following technical problems:

[0004] Due to the randomness and uncertainty of pedestrian motion, future pedestrian trajectories are multi-faceted (i.e., multimodal). Multimodal trajectory prediction is a very challenging task. Existing trajectory prediction methods simplify trajectory multimodality into destination intention multimodality. This approach first predicts multiple trajectory destinations and then expands these multiple predicted destinations into multiple trajectories. This trajectory prediction method ignores intermediate intentions, as pedestrians may choose different intermediate paths to reach the same destination. This results in low accuracy and reliability of the final prediction results. Summary of the Invention

[0005] The present invention aims to provide a multimodal pedestrian trajectory prediction method, system, device, and medium to address one or more of the aforementioned technical problems. This invention addresses the drawback of existing multimodal pedestrian trajectory prediction methods, which ignore intermediate intentions. A multimodal pedestrian trajectory prediction method based on multiple intentions is proposed, which improves the accuracy and reliability of pedestrian trajectory prediction results.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides a multimodal pedestrian trajectory prediction method, comprising the following steps:

[0008] Obtain the observed pedestrian trajectory to be predicted;

[0009] Based on the obtained observed pedestrian trajectory, a pre-trained multimodal pedestrian trajectory prediction model is used to perform future trajectory prediction to obtain a multimodal pedestrian trajectory prediction result;

[0010] The multimodal pedestrian trajectory prediction model includes:

[0011] A feature extraction module includes an observation feature encoder, an intermediate intention feature encoder, and a target intention feature encoder; wherein the observation feature encoder is used to input an observed pedestrian trajectory and perform feature extraction, and output an observed pedestrian trajectory feature; the intermediate intention feature encoder is used to input an intermediate intention and perform feature extraction, and output an intermediate intention feature code; the target intention feature encoder is used to input a target intention and perform feature extraction, and output a target intention feature code;

[0012] A multi-intention memory module, wherein the multi-intention memory module pre-stores observed pedestrian trajectory feature samples and corresponding target intent features and intermediate intent features; wherein the multi-intention memory module is used to input the observed pedestrian trajectory features and obtain, based on similarity matching, observed pedestrian trajectory feature samples that match the input observed pedestrian trajectory features, as well as target intent feature samples and intermediate intent feature samples corresponding to the observed pedestrian trajectory feature samples;

[0013] The intention refinement module is used to input the observed pedestrian trajectory features, the observed pedestrian trajectory feature samples that match the observed pedestrian trajectory features, and the target intention feature samples and intermediate intention feature samples corresponding to the observed pedestrian trajectory feature samples, and perform deviation adjustment to obtain the intention refinement features;

[0014] The decoder is used to input the refined features of the intention and perform decoding processing, and output a multimodal pedestrian trajectory prediction result.

[0015] A further improvement of the method of the present invention is that the multi-intention memory module includes four storage spaces, namely, the observation pedestrian trajectory feature sample storage space M p , intermediate intent feature sample storage space M i , target intention feature sample storage space M g and usage storage space A; where M p 、M i 、M g There is a one-to-one correspondence between A and A. If any information in one of the storage spaces is known, the corresponding storage information in other storage spaces can be found.

[0016] According to the observed pedestrian trajectory characteristics F p , intermediate intention feature F i , target intention feature F g Whether it already exists in the storage space is divided into two storage methods;

[0017] If the above three features already exist in the storage space, only M is updated. p and A, the expression is,

[0018] A[k]←0; k is the index corresponding to any stored information in the storage space;

[0019] If the above three features do not exist in the storage space, update all storage spaces. The expression is,

[0020] It is the index of the maximum value among all the stored information in storage space A.

[0021] A further improvement of the method of the present invention is that in the multi-intention memory module, in the process of obtaining observed pedestrian trajectory feature samples that match the input observed pedestrian trajectory features based on similarity matching, cosine similarity matching is used, and then a preset number of pieces of information are selected.

[0022] A further improvement of the method of the present invention is that, in the intention refinement module, the step of performing deviation adjustment includes:

[0023] The target intention feature sample and the intermediate intention feature sample corresponding to the observed pedestrian trajectory feature sample are connected and high-dimensional feature extraction is performed to obtain a first high-dimensional feature; the deviation between the observed pedestrian trajectory feature and the observed pedestrian trajectory feature sample is obtained and high-dimensional feature extraction is performed to obtain a second high-dimensional feature;

[0024] The first high-dimensional features and the second high-dimensional features are connected and clustered to obtain features after clustering; the features after clustering are connected with the observed pedestrian trajectory features to obtain features after intention refinement.

[0025] A further improvement of the method of the present invention is that, in the intention refinement module, the step of obtaining the deviation between the observed pedestrian trajectory feature and the observed pedestrian trajectory feature sample includes:

[0026] Use MLP network to calculate the observation trajectory feature F p and the matching observed pedestrian trajectory feature sample M p Deviation F of [k] v , the expression is,

[0027]

[0028] Where, is the MLP network, θ V are the learnable parameters of the network;

[0029] Generate the final intention F based on the deviation r , the expression is,

[0030]

[0031] Where, is a linear neural network, θ V are the learnable parameters of the network, is the feature connection operation.

[0032] A further improvement of the method of the present invention is that in the decoder, an LSTM decoder is used to generate the final trajectory The expression is,

[0033]

[0034] Where, f d (·) is the LSTM decoder.

[0035] A further improvement of the method of the present invention is that in the step of training and acquiring the pre-trained multimodal pedestrian trajectory prediction model,

[0036] The L2 distance between the predicted trajectory and the true future trajectory is used as the loss function of the multimodal pedestrian trajectory prediction model, which is expressed as follows:

[0037]

[0038] Where Y is the real future trajectory, is the predicted trajectory, ||||2 is the L2 distance;

[0039] The stored intent features and the adjusted intent features are used as the loss function of the intent refinement module, which is expressed as:

[0040]

[0041] Where, F i is the intermediate intention feature; F g is the target intention feature; F r To generate final intention based on deviation; Feature connection operation.

[0042] A second aspect of the present invention provides a multimodal pedestrian trajectory prediction system, comprising:

[0043] A data acquisition module, used to obtain the observed pedestrian trajectory to be predicted;

[0044] A trajectory prediction module is used to predict future trajectories based on the observed pedestrian trajectories using a pre-trained multimodal pedestrian trajectory prediction model to obtain a multimodal pedestrian trajectory prediction result;

[0045] The multimodal pedestrian trajectory prediction model includes:

[0046] A feature extraction module includes an observation feature encoder, an intermediate intention feature encoder, and a target intention feature encoder; wherein the observation feature encoder is used to input an observed pedestrian trajectory and perform feature extraction, and output an observed pedestrian trajectory feature; the intermediate intention feature encoder is used to input an intermediate intention and perform feature extraction, and output an intermediate intention feature code; the target intention feature encoder is used to input a target intention and perform feature extraction, and output a target intention feature code;

[0047] A multi-intention memory module, wherein the multi-intention memory module pre-stores observed pedestrian trajectory feature samples and corresponding target intent features and intermediate intent features; wherein the multi-intention memory module is used to input the observed pedestrian trajectory features and obtain, based on similarity matching, observed pedestrian trajectory feature samples that match the input observed pedestrian trajectory features, as well as target intent feature samples and intermediate intent feature samples corresponding to the observed pedestrian trajectory feature samples;

[0048] The intention refinement module is used to input the observed pedestrian trajectory features, the observed pedestrian trajectory feature samples that match the observed pedestrian trajectory features, and the target intention feature samples and intermediate intention feature samples corresponding to the observed pedestrian trajectory feature samples, and perform deviation adjustment to obtain the intention refinement features;

[0049] The decoder is used to input the refined features of the intention and perform decoding processing, and output a multimodal pedestrian trajectory prediction result.

[0050] A third aspect of the present invention provides an electronic device, comprising:

[0051] at least one processor; and,

[0052] a memory communicatively connected to the at least one processor; wherein,

[0053] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal pedestrian trajectory prediction method as described in any one of the first aspects of the present invention.

[0054] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the multimodal pedestrian trajectory prediction method described in any one of the first aspects of the present invention.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] In the multimodal pedestrian trajectory prediction method provided by the present invention, in response to the defect that the existing multimodal pedestrian trajectory prediction ignores the intermediate intention, a multimodal pedestrian trajectory prediction method based on multi-intention is specifically proposed, and the trajectory prediction result is obtained through a pre-trained multimodal pedestrian trajectory prediction model; wherein, the intermediate intention and target intention of each modality are stored and retrieved through the multi-intention memory module in the multimodal pedestrian trajectory prediction model, and the stored multi-intention is improved through the intention refinement module, and the improved multi-intention memory is used to guide the generation of different trajectories; based on the above-mentioned multi-intention memory module and intention refinement module, the generated predicted trajectory takes into account the intermediate intention and target intention at the same time, and the multimodal trajectory finally generated can cover more reasonable trajectories and has good accuracy and reliability; at the same time, it also provides a new idea for solving the difficult task of multimodal trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 1 is a flow chart of a multimodal pedestrian trajectory prediction method provided by an embodiment of the present invention;

[0059] Figure 2 1 is a schematic diagram of the structure of a multimodal pedestrian trajectory prediction model in an embodiment of the present invention;

[0060] Figure 3 1 is a schematic structural diagram of a multi-intention memory module according to an embodiment of the present invention;

[0061] Figure 4 1 is a schematic diagram of the training process of the multimodal pedestrian trajectory prediction model in an embodiment of the present invention;

[0062] Figure 5 1 is a schematic diagram of a test process of a multimodal pedestrian trajectory prediction model in an embodiment of the present invention;

[0063] Figure 6 is a schematic diagram of some results on the Stanford Drone Dataset in an embodiment of the present invention;

[0064] Figure 7 1 is a schematic diagram illustrating the principle of trajectory prediction in an embodiment of the present invention;

[0065] Figure 8 Schematic diagram of a multimodal pedestrian trajectory prediction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0067] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0068] The present invention is described in further detail below with reference to the accompanying drawings:

[0069] See also Figure 1 The embodiment of the present invention provides a multimodal pedestrian trajectory prediction method, comprising the following specific steps:

[0070] Step 1: Obtain the observed pedestrian trajectory to be predicted;

[0071] Step 2: Based on the observed pedestrian trajectory obtained in step 1, a pre-trained multimodal pedestrian trajectory prediction model is used to perform future trajectory prediction to obtain a multimodal pedestrian trajectory prediction result.

[0072] See also Figure 2 In an embodiment of the present invention, the multimodal pedestrian trajectory prediction model includes:

[0073] A feature extraction module is configured to input an observed pedestrian trajectory, a target intention, and an intermediate intention, perform feature extraction on each of the observed pedestrian trajectory, the target intention, and the intermediate intention, and output the extracted observed pedestrian trajectory features, the target intention features, and the intermediate intention features; illustratively, the module may include: an observed feature encoder, an intermediate intention feature encoder, and a target intention feature encoder;

[0074] A pre-trained multi-intention memory module is used to input the observed pedestrian trajectory features, obtain the stored observed pedestrian trajectory features, and the target intention features and intermediate intention features corresponding to the stored observed pedestrian trajectory features;

[0075] The intention refinement module is configured to input the target intention feature and the intermediate intention feature corresponding to the stored observed pedestrian trajectory feature, perform concatenation, and extract high-dimensional features to obtain a first high-dimensional feature; input the observed pedestrian trajectory feature and the stored observed pedestrian trajectory feature, perform difference acquisition and high-dimensional feature extraction to obtain a second high-dimensional feature; concatenate and cluster the first high-dimensional feature and the second high-dimensional feature (for the purpose of removing redundancy) to obtain a clustered feature; and concatenate the clustered feature with the observed pedestrian trajectory feature to output a feature after intention refinement.

[0076] The decoder is used to input the refined features of the intention and perform decoding processing to output the predicted pedestrian trajectory.

[0077] An embodiment of the present invention discloses a multimodal pedestrian trajectory prediction method based on multiple intentions. To address the problem that existing methods ignore the intermediate intentions of the predicted trajectory, resulting in the generated multimodal trajectory being unable to cover all possible trajectories, the present method constructs an unsupervised multi-intention memory module to store and retrieve the intermediate intentions and target intentions of each modality, and constructs an intention refinement module to improve the stored multi-intention memory. The improved multi-intention memory is used to guide the generation of different trajectories. The present invention uses a multi-intention memory module and an intention refinement module to enable the generated predicted trajectory to simultaneously consider the intermediate intentions and target intentions, thereby enabling the generated multimodal trajectory to cover more reasonable trajectories; it also provides a new approach to solving the difficult task of multimodal trajectory prediction.

[0078] See also Figure 3 In the embodiment of the present invention, the multi-intention memory module includes four storage spaces, namely, the observed pedestrian trajectory feature sample storage space M p , intermediate intent feature sample storage space M i , target intention feature sample storage space M g and usage storage space A; where M p 、M i 、M g There is a one-to-one correspondence between and A. If any information in one of the storage spaces is known, the corresponding storage information in other storage spaces can be found.

[0079] See also Figure 4 In an embodiment of the present invention, the training and acquisition steps of the pre-trained multimodal pedestrian trajectory prediction model include:

[0080] Step 1: Extract features of pedestrian observation trajectories and intentions to obtain a training sample set;

[0081] Specifically and illustratively, step 1 includes:

[0082] Collect the coordinates of the observed pedestrian trajectory, the midpoint coordinates of the future pedestrian trajectory, and the end point coordinates of the future pedestrian trajectory;

[0083] The collected coordinates are used as input samples, and LSTM is used as the encoder to extract the pedestrian observation trajectory features and intention features;

[0084] Step 2: Use the multi-intention memory module to store the feature information obtained in step 1 and retrieve it: The multi-intention memory module is constructed as follows: Figure 3 As shown; the stored feature information includes: the observed trajectory feature, intermediate intention feature and target intention feature obtained in step 1; during training and testing, the collected observed pedestrian trajectory feature is used to retrieve its corresponding stored observed pedestrian trajectory feature, thereby obtaining the intermediate intention feature and target intention feature that match the stored observed pedestrian trajectory feature;

[0085] Step 3: Use the intent refinement module to adjust the retrieved intent features. The adjustment steps include:

[0086] 3.1) Calculate the observed pedestrian trajectory feature F p Matched with the stored observed pedestrian features M p Deviation of [k];

[0087] 3.2) Generate adjusted intent features based on the deviation;

[0088] Step 4: decoding and generating a pedestrian multimodal predicted trajectory based on the adjusted intention features;

[0089] Step 5: Calculate the loss function of the multimodal pedestrian trajectory prediction network;

[0090] Step 6: Optimize network parameters to generate multimodal pedestrian trajectories, including:

[0091] 6.1) Iteratively optimize the network parameters based on the loss function in step 5;

[0092] 6.2) After reaching the predetermined number of iterations, the pedestrian trajectory prediction network generates multimodal pedestrian prediction trajectories.

[0093] An embodiment of the present invention discloses a multimodal pedestrian trajectory prediction method based on multiple intentions. To address the problem that existing methods ignore intermediate intentions in the process of generating multimodal pedestrian trajectories, the present invention proposes a multi-intent memory module to store and retrieve the intermediate intentions and target intentions of each modality, and proposes an intention refinement module to improve the stored multi-intent memory, thereby generating more comprehensive and accurate multimodal pedestrian trajectories driven by multiple intentions.

[0094] Specifically, an embodiment of the present invention provides a multi-modal pedestrian trajectory prediction method based on multiple intentions, the steps comprising:

[0095] Step 1: Extract features of pedestrian observation trajectory and intention, including:

[0096] Collect the observed trajectory coordinates, future trajectory midpoint coordinates and future trajectory end point coordinates as input to extract the pedestrian observation trajectory and intention features; for a target pedestrian, its observed trajectory is known Where (x t ,y t ) is the 2D coordinate at time t. The purpose of this invention is to predict K possible trajectories. Interpretive, midpoint coordinates End point coordinates

[0097] In order to extract the observed trajectory, the intermediate intention of the future trajectory and the target intention of the future trajectory, the present invention uses three LSTM structure encoders to extract features: F p =f p (X); F i =f i (Y i );F g =f g (Y g );

[0098] Among them, F p 、F i and F g are respectively the observation trajectory features, intermediate intention features and target trajectory features; f p (·),f i (·) and f g (·) are the LSTM encoders corresponding to the observed trajectory features, intermediate intention features, and target trajectory features, respectively.

[0099] Step 2: Use the multi-intention memory module to store and retrieve feature information, including:

[0100] 2.1) Construct a multi-intention memory module, including:

[0101] The multi-intention memory module includes four storage spaces: observation trajectory storage space Mp , intermediate intent storage space M i , target intention storage space M g And the usage storage space A. M p 、M i 、M g There is a one-to-one correspondence between , and A, that is, if any information in one of the storage spaces is known, its corresponding storage information in other storage spaces can be found.

[0102] 2.2) Storing feature information, specifically including: obtaining the observation trajectory feature F according to step 1 p , intermediate intent feature F i , target intention feature F g Whether it already exists in the storage space, there are two storage methods;

[0103] If the above three features already exist in the storage space, only M is updated. p and A:

[0104] A[k]←0; is the index corresponding to any stored information in the storage space;

[0105] If the above three features do not exist in the storage space, update all storage spaces:

[0106] It is the index of the maximum value among all the stored information in storage space A.

[0107] 2.3) Use the observed trajectory features to retrieve the corresponding intermediate intention features and target intention features.

[0108] For example, the observed pedestrian trajectory is used to retrieve the storage space to obtain the intermediate intention and the target intention. First, the observed trajectory feature F p With observation storage space M p Perform cosine similarity matching on all stored information in the , and then select the first K information M p [k], and find out the K information in M i and M g The corresponding K groups of intention information M i [k] and M g [k].

[0109] Step 3: Use the intent refinement module to adjust the retrieved intent features, including:

[0110] 3.1) Calculate the observation trajectory feature F p The observation features M that match p [k] deviation; where the MLP network is used to calculate the observation trajectory feature Fp The observation features M that match p Deviation F of [k] v :

[0111]

[0112] Where, is the MLP network, θ V are the learnable parameters of the network.

[0113] 3.2) Generate adjusted intent features based on the deviation; wherein, the final intent F is generated by the deviation obtained in step 3.1) r :

[0114]

[0115] Where, is a linear neural network, θ V are the learnable parameters of the network, is the feature connection operation.

[0116] Step 4: Generate pedestrian multimodal prediction trajectory, including:

[0117] The final intent feature F obtained in step 3 r , we use LSTM encoder to generate the final trajectory

[0118]

[0119] Where, f d (·) is the LSTM decoder.

[0120] Step 5: Calculate the loss function of the multimodal pedestrian trajectory prediction network:

[0121] 5.1) Introduction to encoder-decoder loss function;

[0122] We use the L2 distance between the predicted trajectory and the true future trajectory as the encoder-decoder loss function:

[0123]

[0124] Where Y is the real future trajectory, is the predicted trajectory, and ||||2 is the L2 distance.

[0125] 5.2) Introduction to the loss function of the intent refinement module;

[0126] The stored intent features and the adjusted intent features are used as the loss function of the intent refinement module:

[0127]

[0128] Where, F i is the intermediate intention feature, F g is the target intention feature, F r To generate the final intent based on the deviation, Feature connection operation.

[0129] Step 6: Optimize network parameters to generate multimodal pedestrian trajectories, including:

[0130] 6.1) Iteratively optimize the network parameters based on the loss function obtained in step 4; use the Adam optimizer for 100 iterations with a learning rate of 0.0005.

[0131] 6.2) After reaching the predetermined number of iterations, the pedestrian trajectory prediction network generates multimodal pedestrian prediction trajectories.

[0132] See also Figure 5 After the iteration is completed, according to Figure 5 Test, under the condition of inputting only one observation trajectory, by indexing K desired intent information in the trained multi-intention memory module, and generating the final K multimodal trajectories through the intent refinement module and the trajectory generation module LSTM decoder.

[0133] In summary, the method of the present invention introduces an LSTM encoder, a feature extraction module, to extract features from an observed trajectory in a real-world scenario. K desired intents are indexed in a trained multi-intent memory module, the intent refinement module obtains the adjusted intents, and the trajectory generation module, an LSTM decoder, generates the final K multimodal trajectories. This method undergoes qualitative and quantitative comparative experimental analysis with existing methods, verifying its effectiveness on two public datasets: the ETH / UCY and Stanford Drone Dataset.

[0134] Tables 1a and 1b are respectively the quantitative experimental results of the embodiments of the present invention. Table 1a is the experimental result of the method under the ETH / UCY dataset, and Table 1b is the experimental result of the method under the Stanford Drone Dataset.

[0135] Table 1a. ADE / FDE experimental results of the method on the ETH / UCY dataset

[0136]

[0137] Table 1b. ADE / FDE experimental results of this method on the Stanford Drone Dataset

[0138]

[0139] ADE and FDE are commonly used metrics for measuring trajectory prediction error. Smaller values ​​indicate lower prediction error and better performance. ADE (Average Distance Error) is the average distance error (L2) between the predicted and true trajectory points. FDE (Final Distance Error) is the final distance error (L2) between the predicted and true trajectory points.

[0140] As shown in Table 1a, on the ETH / UCY dataset, the trajectories generated using the method of the present invention achieve the best average performance across the five sub-datasets in terms of the ADE / FDE indicators. As shown in Table 1b, on the Stanford DroneDataset, the predicted trajectories generated by the method of the present invention achieve the best performance in terms of both ADE and FDE indicators. Therefore, from a quantitative perspective, the multi-intention-based multimodal pedestrian trajectory prediction method disclosed in the present invention can generate more accurate predicted trajectories.

[0141] See also Figure 6 , Figure 6 It is the qualitative test result of the present invention. Figure 6 This is a comparison of the predicted trajectory generated by the embodiment of the present invention and the previous method PECNet based on the Stanford Drone Dataset data. Compared with the original method PECNet, the predicted trajectory generated by the embodiment of the present invention is more accurate. For example, in the Bookstore and Little scenarios, the trajectory predicted by the method of the present invention is very close to the true trajectory, while the trajectory predicted by the original method PECNet has a large deviation from the true trajectory; in the Hyang and Coupa scenarios, the method of the embodiment of the present invention and the original method PECNet both predicted similar terminal trajectories, but the method of the embodiment of the present invention better fits the intermediate process of the trajectory, which also shows the effectiveness of the use of multiple intentions (intermediate intentions and target intentions) in the present invention. Therefore, from the qualitative results, the multimodal pedestrian trajectory prediction method based on multiple intentions of the present invention can generate more accurate predicted trajectories.

[0142] The following are device embodiments of the present invention, which can be used to perform the method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0143] See also Figure 7 and Figure 8 , an embodiment of the present invention discloses a multimodal pedestrian trajectory prediction system, comprising:

[0144] A data acquisition module, used to obtain the observed pedestrian trajectory to be predicted;

[0145] A trajectory prediction module is used to predict future trajectories based on the observed pedestrian trajectories using a pre-trained multimodal pedestrian trajectory prediction model to obtain a multimodal pedestrian trajectory prediction result;

[0146] The multimodal pedestrian trajectory prediction model includes:

[0147] A feature extraction module includes an observation feature encoder, an intermediate intention feature encoder, and a target intention feature encoder; wherein the observation feature encoder is used to input an observed pedestrian trajectory and perform feature extraction, and output an observed pedestrian trajectory feature; the intermediate intention feature encoder is used to input an intermediate intention and perform feature extraction, and output an intermediate intention feature code; the target intention feature encoder is used to input a target intention and perform feature extraction, and output a target intention feature code;

[0148] A multi-intention memory module, wherein the multi-intention memory module pre-stores observed pedestrian trajectory feature samples and corresponding target intent features and intermediate intent features; wherein the multi-intention memory module is used to input the observed pedestrian trajectory features and obtain, based on similarity matching, observed pedestrian trajectory feature samples that match the input observed pedestrian trajectory features, as well as target intent feature samples and intermediate intent feature samples corresponding to the observed pedestrian trajectory feature samples;

[0149] The intention refinement module is used to input the observed pedestrian trajectory features, the observed pedestrian trajectory feature samples that match the observed pedestrian trajectory features, and the target intention feature samples and intermediate intention feature samples corresponding to the observed pedestrian trajectory feature samples, and perform deviation adjustment to obtain the intention refinement features;

[0150] The decoder is used to input the refined features of the intention and perform decoding processing, and output a multimodal pedestrian trajectory prediction result.

[0151] The system of the present invention addresses the problem that the intermediate intentions of pedestrians are currently ignored in the multimodal pedestrian trajectory prediction process, and the generated multimodal trajectories cannot cover all possible trajectories. It introduces a multimodal pedestrian trajectory prediction method based on multiple intentions to more comprehensively generate possible future intentions; it uses a pedestrian trajectory prediction network based on a multi-intention memory module and an intention refinement module to generate more accurate possible future trajectories.

[0152] In summary, embodiments of the present invention disclose a multimodal pedestrian trajectory prediction method and system based on multiple intentions, which lies at the intersection of computer vision and autonomous driving. The present invention generates multiple possible predicted trajectories for a single observed pedestrian trajectory in a real-world scenario. Pedestrian trajectory prediction is crucial in many visual tasks, but is challenging due to the multimodality of future trajectories. Most existing methods model multimodality through target intent, but ignore beneficial intermediate intents, which hinders performance improvement. To address this issue, the present invention proposes a multi-intention-based memory storage network for pedestrian trajectory prediction, which uses a multi-intention mechanism of intermediate intents (intermediate states) and target intents (destination) to model multimodality. Specifically, the present invention introduces an LSTM neural network as an encoder-decoder network. The encoder extracts observed trajectory features, intermediate intent features, and target intent features, while the decoder generates predicted trajectories. A multi-intention memory module is used to store and retrieve intermediate intent features and destination intent features corresponding to the observed trajectory features. An intent refinement module is introduced to adjust the intent features retrieved from the multi-intention module. Finally, the observed trajectory features in the encoder and the intent features obtained in the intent refinement module are used as input to the decoder, ultimately resulting in a predicted pedestrian trajectory.

[0153] In another embodiment of the present invention, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the multimodal pedestrian trajectory prediction method.

[0154] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the multimodal pedestrian trajectory prediction method in the above embodiment.

[0155] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0156] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0157] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0158] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A multimodal pedestrian trajectory prediction method, characterized in that: The following steps are involved: Obtain the observed pedestrian trajectory to be predicted; Based on the obtained observed pedestrian trajectory, a pre-trained multimodal pedestrian trajectory prediction model is used to perform future trajectory prediction to obtain a multimodal pedestrian trajectory prediction result; The multimodal pedestrian trajectory prediction model includes: A feature extraction module includes an observation feature encoder, an intermediate intention feature encoder, and a target intention feature encoder; wherein the observation feature encoder is used to input an observed pedestrian trajectory and perform feature extraction, and output an observed pedestrian trajectory feature; the intermediate intention feature encoder is used to input an intermediate intention and perform feature extraction, and output an intermediate intention feature code; the target intention feature encoder is used to input a target intention and perform feature extraction, and output a target intention feature code; A multi-intention memory module, wherein the multi-intention memory module pre-stores observed pedestrian trajectory feature samples and corresponding target intent features and intermediate intent features; wherein the multi-intention memory module is used to input the observed pedestrian trajectory features and obtain, based on similarity matching, observed pedestrian trajectory feature samples that match the input observed pedestrian trajectory features, as well as target intent feature samples and intermediate intent feature samples corresponding to the observed pedestrian trajectory feature samples; The intention refinement module is used to input the observed pedestrian trajectory features, the observed pedestrian trajectory feature samples that match the observed pedestrian trajectory features, and the target intention feature samples and intermediate intention feature samples corresponding to the observed pedestrian trajectory feature samples, and perform deviation adjustment to obtain the intention refinement features; The decoder is used to input the refined features of the intention and perform decoding processing, and output a multimodal pedestrian trajectory prediction result.

2. A multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: The multi-intention memory module includes four storage spaces, namely, the observation pedestrian trajectory feature sample storage space M p , intermediate intent feature sample storage space M i , target intention feature sample storage space M g and usage storage space A; where M p 、M i 、M g There is a one-to-one correspondence between A and A. If any information in one of the storage spaces is known, the corresponding storage information in other storage spaces can be found. According to the observed pedestrian trajectory characteristics F p , intermediate intention feature F i , target intention feature F g Whether it already exists in the storage space is divided into two storage methods; If the above three features already exist in the storage space, only M is updated. p and A, the expression is, A[k]←0; k is the index corresponding to any stored information in the storage space; If the above three features do not exist in the storage space, update all storage spaces. The expression is, It is the index of the maximum value among all the stored information in storage space A.

3. The multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: In the multi-intention memory module, in the process of obtaining observed pedestrian trajectory feature samples that match the input observed pedestrian trajectory features based on similarity matching, cosine similarity matching is adopted, and then a preset number of pieces of information are selected.

4. The multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: In the intention refinement module, the step of performing deviation adjustment includes: The target intention feature sample and the intermediate intention feature sample corresponding to the observed pedestrian trajectory feature sample are connected and high-dimensional feature extraction is performed to obtain a first high-dimensional feature; the deviation between the observed pedestrian trajectory feature and the observed pedestrian trajectory feature sample is obtained and high-dimensional feature extraction is performed to obtain a second high-dimensional feature; The first high-dimensional features and the second high-dimensional features are connected and clustered to obtain features after clustering; the features after clustering are connected with the observed pedestrian trajectory features to obtain features after intention refinement.

5. The multimodal pedestrian trajectory prediction method according to claim 4, characterized in that: In the intention refinement module, the step of obtaining the deviation between the observed pedestrian trajectory feature and the observed pedestrian trajectory feature sample includes: Use MLP network to calculate the observation trajectory feature F p and the matching observed pedestrian trajectory feature sample M p Deviation F of [k] v , the expression is, Where, is the MLP network, θ V are the learnable parameters of the network; Generate the final intention F based on the deviation r , the expression is, Where, is a linear neural network, θ V are the learnable parameters of the network, is the feature connection operation.

6. A multimodal pedestrian trajectory prediction method according to claim 5, characterized in that: In the decoder, the LSTM decoder is used to generate the final trajectory The expression is, Where, f d (·) is the LSTM decoder.

7. The multimodal pedestrian trajectory prediction method according to claim 1, characterized in that: In the step of training and acquiring the pre-trained multimodal pedestrian trajectory prediction model, The L2 distance between the predicted trajectory and the true future trajectory is used as the loss function of the multimodal pedestrian trajectory prediction model, which is expressed as follows: Where Y is the real future trajectory, is the predicted trajectory, ||||2 is the L2 distance; The stored intent features and the adjusted intent features are used as the loss function of the intent refinement module, which is expressed as: Where, F i is the intermediate intention feature; F g is the target intention feature; F r To generate final intention based on deviation; Feature connection operation.

8. A multimodal pedestrian trajectory prediction system, characterized in that: include: A data acquisition module, used to obtain the observed pedestrian trajectory to be predicted; A trajectory prediction module is used to predict future trajectories based on the observed pedestrian trajectories using a pre-trained multimodal pedestrian trajectory prediction model to obtain a multimodal pedestrian trajectory prediction result; The multimodal pedestrian trajectory prediction model includes: A feature extraction module includes an observation feature encoder, an intermediate intention feature encoder, and a target intention feature encoder; wherein the observation feature encoder is used to input an observed pedestrian trajectory and perform feature extraction, and output an observed pedestrian trajectory feature; the intermediate intention feature encoder is used to input an intermediate intention and perform feature extraction, and output an intermediate intention feature code; the target intention feature encoder is used to input a target intention and perform feature extraction, and output a target intention feature code; A multi-intention memory module, wherein the multi-intention memory module pre-stores observed pedestrian trajectory feature samples and corresponding target intent features and intermediate intent features; wherein the multi-intention memory module is used to input the observed pedestrian trajectory features and obtain, based on similarity matching, observed pedestrian trajectory feature samples that match the input observed pedestrian trajectory features, as well as target intent feature samples and intermediate intent feature samples corresponding to the observed pedestrian trajectory feature samples; The intention refinement module is used to input the observed pedestrian trajectory features, the observed pedestrian trajectory feature samples that match the observed pedestrian trajectory features, and the target intention feature samples and intermediate intention feature samples corresponding to the observed pedestrian trajectory feature samples, and perform deviation adjustment to obtain the intention refinement features; The decoder is used to input the refined features of the intention and perform decoding processing, and output a multimodal pedestrian trajectory prediction result.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the multimodal pedestrian trajectory prediction method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the multimodal pedestrian trajectory prediction method according to any one of claims 1 to 7 is implemented.

Images