A cross-scene adaptive pedestrian trajectory prediction method and device

Abstract

The application discloses a cross-scene adaptive pedestrian trajectory prediction method and device, and the method comprises the following steps: S11, converting source domain real trajectory data into target domain aligned trajectory data, and converting target domain real trajectory data into source domain aligned trajectory data; S12, respectively calculating the authenticity scores of the target domain aligned trajectory data, the target domain real trajectory data, the source domain aligned trajectory data and the source domain real trajectory data; S13, combining the authenticity scores to calculate a generation loss to obtain an optimal generation model; S21, sampling target domain data from a target domain data set, and the optimal generation model generates a future prediction trajectory of a pedestrian according to the target domain data, which is used for a cross-scene prediction task of an autonomous mobile robot or an automatic driving vehicle. The application can make the data level complete effective knowledge migration, make the optimal generation model learn target domain information, improve the adaptability of the model in the target domain, and thus solve the model adaptability problem caused by scene differences.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pedestrian trajectory prediction, and particularly relates to a cross-scene adaptive pedestrian trajectory prediction method and device. BACKGROUND

[0002] With the rapid development of intelligent manufacturing technology and intelligent systems, the importance of human-computer interaction technology in the new round of technological revolution is increasing. As a core part of human-computer interaction system, pedestrian trajectory prediction plays a key role in application scenarios such as autonomous driving cars and self-navigating robots. Pedestrian trajectory prediction aims to predict the future trajectory of a pedestrian according to the observed historical trajectory of the pedestrian, and usually involves modeling and reasoning the motion of pedestrians in different scenarios. Due to the inherent randomness of human movement intention, the complex mutual influence between pedestrians, and the difference between different application scenarios, high-precision and generalizable pedestrian trajectory prediction is still a very challenging problem.

[0003] In recent years, due to the outstanding performance of neural networks in computer vision, natural language processing and other fields, data-driven deep learning methods have become the mainstream of pedestrian trajectory prediction technology. Since the pedestrian trajectory is essentially a time series, recurrent neural networks (RNN) are used to extract trajectory time series features to generate future trajectories with similar features; to extract the complex spatial-social interaction features between pedestrians, graph neural networks (GNN) model the scene as a graph and extract the spatial relationship between pedestrian nodes; considering the diversity of pedestrian movement intention, some methods based on conditional variational autoencoder (CVAE) generate multiple reasonable trajectories by introducing a random sampling process in the output stage, while the methods based on generative adversarial network (GAN) directly output multiple trajectories with social affinity through the generative model. For example, the existing technology one discloses a pedestrian trajectory prediction method and system through Transformer to complete attention-based time series feature extraction; the existing technology two discloses an automatic driving pedestrian trajectory prediction method and device through fusion of interaction features between pedestrians to complete prediction.

[0004] Although the above methods have contributed to improving the prediction accuracy to varying degrees, they only consider the test performance of the prediction model in a single scene, and assume that pedestrians have a unified motion pattern in different scenes, ignoring the huge differences between different scenes. At the same time, the phenomenon of scene difference is very common in the fields of autonomous driving and self-navigation. For example, an autonomous driving car that has just driven on a suburban highway enters an urban street, and the complexity of scene factors such as the number of pedestrians and obstacle density will also increase. And this increase in scene complexity inevitably requires the vehicle-mounted trajectory prediction system to maintain high stability and reliability in different scenes to ensure that the deviation of the model prediction does not become a safety hazard.

[0005] In order to directly apply the pedestrian trajectory prediction method to a real product (such as an autonomous vehicle), the model adaptability problem caused by scene differences must be solved. SUMMARY

[0006] The purpose of the present application is to solve the problem of poor adaptability of the prior art to different domains (scenes) when predicting pedestrian trajectories, and to provide a cross-scene adaptive pedestrian trajectory prediction method and device.

[0007] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0008] A cross-scene adaptive pedestrian trajectory prediction method, comprising the following steps: S11: converting source domain real trajectory data into target domain aligned trajectory data, and converting target domain real trajectory data into source domain aligned trajectory data; S12: calculating the authenticity scores of the target domain aligned trajectory data, target domain real trajectory data, source domain aligned trajectory data and source domain real trajectory data, respectively; S13: combining the authenticity scores of the target domain aligned trajectory data and the authenticity scores of the target domain real trajectory data to calculate the target domain generation loss, and combining the authenticity scores of the source domain aligned trajectory data and the authenticity scores of the source domain real trajectory data to calculate the source domain generation loss, and iteratively updating to obtain an optimal generation model; S21: sampling target domain data from a target domain data set, and the optimal generation model generates a future predicted trajectory of a pedestrian according to the target domain data, thereby being used for a cross-scene prediction task of an autonomous mobile robot or an autonomous vehicle.

[0009] In some embodiments of the present application, in step S13, the optimal generation model also generates source domain aligned trajectory data, mixes the source domain aligned trajectory data with source domain real trajectory data to obtain enhanced source domain data; divides the enhanced source domain data into enhanced source domain historical observation data and source domain future true value data, and divides the target domain real trajectory data into target domain historical observation data and target domain future true value data.

[0010] In some embodiments of the present application, the following steps are further included: S14: mapping the enhanced source domain historical observation data and the target domain historical observation data to the same high-dimensional feature space, and respectively generating enhanced source domain historical observation data intermediate features and target domain historical observation data intermediate features participating in domain alignment, and then calculating the authenticity scores of the enhanced source domain historical observation data intermediate features and the authenticity scores of the target domain historical observation data intermediate features; S15: training an optimal generation model comprising an optimal encoder and an optimal decoder by combining the authenticity scores of the source domain historical observation data intermediate features and the authenticity scores of the target domain historical observation data intermediate features, wherein the optimal encoder outputs domain-invariant features that are common to different domains, and the optimal decoder generates a pedestrian predicted trajectory according to the domain-invariant features.

[0011] In some embodiments of the present application, in step S11, a target domain generator and a source domain generator are respectively established, the target domain generator converts the source domain real trajectory data into the target domain aligned trajectory data, the input of the target domain generator is the source domain real trajectory data, the source domain generator converts the target domain real trajectory data into the source domain aligned trajectory data, the target domain generator first performs continuous time convolution on the time dimension through a time convolution network to obtain an intermediate tensor of the source domain trajectory data, the time convolution network has a multi-head attention layer for outputting an expression tensor processed by an attention mechanism, and the expression tensor is spliced with a feature dimension, and a full connection layer is used to adjust the dimension of the feature dimension to be the same as the dimension of the time dimension.

[0012] In some embodiments of the present application, in step S12, a source domain discriminator and a target domain discriminator are respectively established, the target domain discriminator is used to derive the authenticity scores of the target domain aligned trajectory data and the target domain real trajectory data, the source domain discriminator is used to derive the authenticity scores of the source domain aligned trajectory data and the source domain real trajectory data, and the authenticity scores of the target domain aligned trajectory data and the target domain real trajectory data are calculated by the following formula:

[0013]

[0014] wherein, I ST represents the authenticity score of the target domain aligned trajectory data, I TT represents the authenticity score of the target domain real trajectory data, X ST represents the target domain aligned trajectory data, X T represents the target domain real trajectory data, D T represents the target domain discriminator, and W dt represents all learnable parameters in the target domain discriminator.

[0015] In some embodiments of the present application, in step S13, the following steps are included: S131: source domain real trajectory data and target domain real trajectory data are respectively sampled from a source domain data set and a target domain data set; S132: a data level alignment module is constructed and iteratively trained, and a first target function is calculated each time:

[0016]

[0017] wherein, represents the total loss function of the data level alignment module, X S represents the source domain real trajectory data, represents the target domain generation loss, a representative source domain generation loss, a representative cycle consistency loss, a representative ontology loss, a is a weight coefficient, and β is a weight coefficient;

[0018] S133: The gradients of the neural network are calculated and the parameters of the target domain generator, the source domain generator, the source domain discriminator, and the target domain discriminator are updated respectively to obtain an optimal generation model, and the expression of the optimal generation model is as follows:

[0019]

[0020] wherein G S→T represents the target domain generator, G T→S represents the source domain generator, D S represents the source domain discriminator, and D T represents the target domain discriminator.

[0021] In some embodiments of the present application, step S14 comprises constructing a feature-level alignment embedding discriminator; the feature-level alignment embedding discriminator outputs a true or false score of the source domain historical observation data intermediate feature and a true or false score of the target domain historical observation data intermediate feature according to the enhanced source domain historical observation data intermediate feature and the target domain historical observation data intermediate feature; the network structure of the feature-level alignment embedding discriminator is at least three layers, and the output of the nth layer is:

[0022]

[0023] wherein r (n) represents the output tensor of the nth layer, the first two layers of the network structure are followed by batch normalization layers, and the network structure finally outputs a true or false score I S of the source domain historical observation data intermediate feature and a true or false score I T of the target domain historical observation data intermediate feature:

[0024]

[0025] wherein D E represents the feature-level alignment embedding discriminator, W e represents all learnable parameters in the feature-level alignment embedding discriminator, h S represents the enhanced source domain historical observation data intermediate feature, and h T represents the target domain historical observation data intermediate feature.

[0026] In some embodiments of the present application, step S15 comprises: constructing and iteratively training a feature-level alignment module, and calculating a second objective function each time iteration is performed:

[0027]

[0028] wherein, represents a total loss function of the feature-level alignment module, represents a prediction loss of the trajectory, represents an adversarial loss of the discriminator, and γ is a weight coefficient of the adversarial loss; the input of the feature-level alignment module is the enhanced source domain historical observation data, the target domain historical observation data, and the source domain future true value, and the output is the optimal encoder and the optimal decoder, and the training target of the feature alignment module is represented as:

[0029]

[0030] wherein, Enc * represents the optimal encoder, and Dec * represents the optimal decoder.

[0031] In some embodiments of the present application, step S21 comprises: calculating a target domain data intermediate feature according to the target domain data, and predicting a future prediction trajectory of the pedestrian according to the target domain data intermediate feature.

[0032] The present application also provides a cross-scene adaptive pedestrian trajectory prediction device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the cross-scene adaptive pedestrian trajectory prediction method according to any one of the above when executing the computer program.

[0033] The present application has the following beneficial effects:

[0034] The cross-scene adaptive pedestrian trajectory prediction method provided by the present application can convert source domain trajectory data into target domain trajectory data to obtain aligned trajectory data, calculate a true and false score of the aligned trajectory data and a true and false score of target domain real trajectory data to obtain an optimal generation model, and generate a future prediction trajectory of a pedestrian after sampling target domain data from a target domain data set using the optimal generation model, so that effective knowledge transfer can be completed in the form of preprocessing at the data level, the optimal generation model learns more target domain information, the adaptability of the optimal generation model to the target domain is improved, the model adaptability problem caused by scene differences is solved, and the method has universality.

[0035] In addition, in some embodiments, the present application also has the following beneficial effects:

[0036] By establishing a feature level alignment module and a feature level alignment discriminator, and then training an optimal generation model by using a large amount of source domain data and a small amount of target domain data, a double alignment domain adaptation scheme can be realized. For the macroscopic difference problem existing between scenes, a domain alignment method oriented to the data level of pedestrian trajectory data in different scenes is used, which can convert the pedestrian trajectory domain of the source domain to the target domain. For the microscopic difference problem existing between scenes, a domain alignment method at the feature level of pedestrian trajectory is used, which can further transfer the source domain knowledge to the target domain, improve the ability of the optimal generation model to generate domain general knowledge, thereby improving the prediction accuracy of the optimal generation model in the target domain and enhancing the adaptability of the model to different domains (scenes).

[0037] By using multiple objective functions to jointly optimize the domain alignment process, the data alignment link is more stable and reliable.

[0038] By using a time convolution network to realize the encoding and decoding of the input trajectory, the convolution operation can aggregate the information of adjacent time nodes, thereby effectively avoiding the error accumulation problem of the recurrent neural network.

[0039] Other beneficial effects in the embodiments of the present application will be further described below. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a schematic diagram of a generative adversarial network in the embodiments of the present application;

[0041] Figure 2 is a schematic diagram of a double alignment domain method in the embodiments of the present application;

[0042] Figure 3 is a network structure diagram of a data level alignment generator in embodiment 1;

[0043] Figure 4 is a network structure diagram of a data level alignment discriminator in embodiment 1;

[0044] Figure 5 is a network structure diagram of a feature level alignment embedding discriminator in embodiment 1;

[0045] Figure 6 is a step flow chart of a cross-scene self-adaptive pedestrian trajectory prediction method in the embodiments of the present application. DETAILED DESCRIPTION

[0046] The present application will be further described below by comparing the drawings and combining the preferred embodiments. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0047] It should be noted that the left, right, up, down, top, bottom and other orientation terms in the embodiment are only relative concepts or references to the normal use state of the product, and should not be considered as limiting.

[0048] As described above, in order to directly apply the pedestrian trajectory prediction method to real products (such as autonomous vehicles), the model adaptability problem caused by scene differences must be solved. How to provide a method to improve the adaptability of the model to different scenes under the condition of a large amount of source domain (familiar scene) data and a small amount of target domain (unfamiliar scene) data, so that the trained model continues to maintain excellent performance in a large amount of target domain data test, is a new challenge and problem to be solved.

[0049] To solve the above problems, the following embodiments of the present application propose a cross-scene adaptive pedestrian trajectory prediction method, referring to Figure 6 , comprising the following steps: S11: converting source domain real trajectory data into target domain aligned trajectory data, and converting target domain real trajectory data into source domain aligned trajectory data; S12: calculating the authenticity scores of the target domain aligned trajectory data, target domain real trajectory data, source domain aligned trajectory data and source domain real trajectory data, respectively;

[0050] S13: combining the authenticity scores of the target domain aligned trajectory data and the authenticity scores of the target domain real trajectory data to calculate the target domain generation loss, and combining the authenticity scores of the source domain aligned trajectory data and the authenticity scores of the source domain real trajectory data to calculate the source domain generation loss, and iteratively updating to obtain an optimal generation model; S21: sampling target domain data from a target domain data set, and the optimal generation model generates a future predicted trajectory of a pedestrian according to the target domain data, thereby being used for a cross-scene prediction task of an autonomous mobile robot or an autonomous vehicle.

[0051] Preferably, in step S13, the optimal generation model also generates source domain aligned trajectory data, mixes the source domain aligned trajectory data with source domain real trajectory data to obtain enhanced source domain data; divides the enhanced source domain data into enhanced source domain historical observation data and source domain future true value data, and divides the target domain real trajectory data into target domain historical observation data and target domain future true value data.

[0052] Preferably, the method further comprises the following steps: S14: mapping the enhanced source domain historical observation data and the target domain historical observation data to the same high-dimensional feature space, and respectively generating enhanced source domain historical observation data intermediate features and target domain historical observation data intermediate features participating in domain alignment, and then calculating a true and false score of the enhanced source domain historical observation data intermediate features and a true and false score of the target domain historical observation data intermediate features; and S15: training an optimal generative model comprising an optimal encoder and an optimal decoder by combining the true and false score of the source domain historical observation data intermediate features and the true and false score of the target domain historical observation data intermediate features, wherein the optimal encoder outputs domain-invariant features that are common to different domains, and the optimal decoder generates a pedestrian prediction trajectory according to the domain-invariant features.

[0053] The embodiment of the present application also provides a cross-scene adaptive pedestrian trajectory prediction device, comprising a processor, a memory and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the cross-scene adaptive pedestrian trajectory prediction method as described above when executing the computer program.

[0054] The embodiment of the present application faces the cross-scene application demand of the model (i.e. the optimal generative model comprising the optimal encoder and the optimal decoder) of the pedestrian trajectory prediction of the real product, and proposes a cross-scene adaptive pedestrian trajectory prediction method, which can be divided into two parts of data-level alignment and feature-level alignment, and the two alignment modes can be used separately or used in combination as a whole scheme, i.e. a double-alignment domain adaptation method.

[0055] The data-level alignment method proposed in the embodiment of the present application is used to complete the data-level alignment of the source domain and the target domain, and based on the source domain real trajectory data generate target domain alignment trajectory data with target domain characteristics wherein N represents the total number of pedestrians in a certain period, t full =t obs +t pred is the total length of the trajectory, including the historical observation length t obs and the future prediction length t pred .

[0056] On the data level of the trajectory domain, there are not only explicit macro differences, but also implicit macro differences. However, these implicit differences are difficult to eliminate directly by the inter-domain distance defined by artificial, and a dynamic method needs to be introduced to learn to narrow the distribution between two different domains. Therefore, the embodiment of the present application adopts a domain adaptation strategy based on a generative adversarial network, and arranges a pair of generators and discriminators to form a GAN network on the data level, so as to align the data from the source domain to the target domain, complete the domain knowledge migration from the source domain to the target domain, and the embodiment of the present application is called a target domain generator GAN.S→T As shown in the accompanying drawings Figure 1 The target domain generator G S→T aims to generate target domain aligned trajectory data X ST with target domain characteristics, while the target domain discriminator D T strives to distinguish the generated target domain aligned trajectory data X ST from the target domain real trajectory data X .

[0057] The generator and the discriminator form a kind of confrontation, and are jointly reinforced in this confrontation. The target domain generator G S→T will reduce the difference between the target domain aligned trajectory data X ST and the target domain real trajectory data X T as much as possible in the training process, so that the target domain discriminator D T cannot distinguish between the generated sample (i.e. the target domain aligned trajectory data X ST ) and the real sample (the target domain real trajectory data X T ); while the target domain discriminator D T will try to determine X ST as a fake sample and X T as a true sample as much as possible, so as to improve its own discrimination ability. The target domain generation loss generated in this confrontation process will guide the continuous optimization of the two.

[0058] The embodiment of the present application also proposes a feature-level alignment method, which is based on the enhanced source domain historical data X' S and the target domain historical data X' T to complete the feature-level alignment of the source domain and the target domain; at the same time, based on the enhanced source domain historical data X' S and the enhanced source domain future data Y S , the training of the prediction network is completed, and accurate prediction trajectory data Y

[0059] Feature-level alignment is a domain alignment operation performed in the high-dimensional feature space of trajectories. Unlike data-level alignment, which only needs to consider the data-level differences between the two domains and can be developed independently of the prediction network, the feature-level alignment module can be embedded in the prediction network. When the feature-level alignment module is embedded in the prediction network, compatibility with the original network must be considered. Existing methods do not consider data-level alignment and use inter-domain distances to complete feature-level alignment. For example, T-GNN uses an attention module added to the intermediate feature layer of the prediction network to obtain domain feature representation, and finally uses the Euclidean distance between the intermediate features of the source domain and the target domain as the alignment loss; while AML adds an RNN network to the output feature layer of the prediction network to learn the target domain knowledge online, and directly reduces the distribution distance between the predicted trajectory and the real trajectory by kernel density estimation. In summary, existing methods statically follow the artificially set training mode, while dynamic domain adaptation can better cope with various differences in complex trajectory domains. The embodiment of the present application uses a discriminator to measure the implicit microscopic differences between the source domain features and the target domain features, and attempts to gradually reduce these differences during training. Since each advanced pedestrian trajectory prediction model currently uses an encoding-decoding structure, i.e., the prediction network is composed of an encoder responsible for generating intermediate hidden variables, and a decoder responsible for deriving future trajectories from hidden variables, the features at the junction of the encoder and the decoder have the greatest domain representation capability. Therefore, the embodiment of the present application adds an adversarial module between the encoder and the decoder, which is responsible for calculating the adversarial loss, and then completes the feature-level domain alignment through adversarial training. The overall framework is shown in the feature-level alignment module of E . Figure 2 .

[0060] Embodiment 1

[0061] This embodiment mainly faces the cross-complex scene application of real intelligent navigation products, such as cross-scene prediction tasks, navigation of autonomous mobile robots or autonomous vehicles, and proposes a double-alignment domain adaptation method for cross-scene adaptive pedestrian trajectory prediction.

[0062] The specific process of this embodiment is described as follows:

[0063] Step S11: Establish a data-level alignment generator for a data-level alignment module. The data-level alignment generator can be divided into a target domain generator GAN S→T and a source domain generator GAN T→S . The target domain generator GAN S→T converts the source domain real trajectory data into target domain aligned trajectory data. The source domain generator GAN T→SThe target domain data is converted into the source domain data, i.e., the target domain real trajectory data is converted into the source domain aligned trajectory data, and the target domain real trajectory data is converted into the source domain aligned trajectory data is an auxiliary process, i.e., a process of participating in model training (for generating an optimal generation model) but not participating in model inference (using the optimal generation model to obtain aligned data); the target domain generator GAN S→T The input of the target domain generator GAN S→T is the source domain real trajectory data.

[0064] Taking the target domain generator GAN S→T as an example, it is responsible for converting the source domain data into the target domain data, and the specific network structure is as shown in Figure 3 The source domain real trajectory data is taken as the input, where N represents the total number of pedestrians in a certain period of time, t full =t obs +t pred is the total length of the trajectory, including the historical observation length t obs and the future prediction length t pred . First, a continuous time convolution operation is performed on the time dimension of the input to obtain an intermediate tensor with a size of N×64×t full . The nth layer of the time convolution network can be represented as:

[0065]

[0066] where ReLU is a nonlinear activation function, TCN is a time convolution operation, is the learnable parameter of the nth layer network. Four layers of time convolution layers TCN are used in this embodiment. In order to further eliminate the implicit social interaction difference, a graph attention module is designed to extract the social-spatial relationship between pedestrians. Specifically, after the second layer of TCN, a multi-head attention layer is inserted to output an expression tensor processed by the attention mechanism:

[0067] c attn =GAT(c (2) ;W gat )

[0068] Then, c attn and the trajectory feature c (2) output by the second layer are spliced in the feature dimension to generate a trajectory feature representation c' attn with social interaction characteristics, and then a fully connected layer FC is used to reduce the feature dimension to 64 dimensions. Then, the remaining two TCN layers reduce the feature dimension from 64 dimensions to 2 dimensions, and the final trajectory feature c (4) is the final output of the target domain generator GAN S→T i.e., target domain aligned trajectory data X ST .

[0069] Step S12: Establishing a data-level alignment discriminator of the data-level alignment module, the data-level alignment discriminator is divided into a target domain discriminator and a source domain discriminator, the target domain discriminator is used to derive the authenticity score of the target domain aligned trajectory data and the authenticity score of the target domain real trajectory data, and the source domain discriminator is responsible for deriving the authenticity score of the source domain aligned trajectory data and the authenticity score of the source domain real trajectory data, and the authenticity score of the source domain aligned trajectory data and the authenticity score of the source domain real trajectory data are also used as auxiliary processes.

[0070] For the target domain discriminator D T , it is responsible for distinguishing the target domain aligned trajectory data X S→T generated by the target domain generator G ST from the target domain real trajectory data X T , and deriving the corresponding authenticity score through binary classification.

[0071] The specific network structure is shown in Figure 4 . Similar to the target domain generator, a four-layer TCN network is also used to build the target domain discriminator D T . However, unlike the target domain generator directly outputting two-dimensional trajectory data, the final output of the target domain discriminator D T is a one-dimensional convolutional sample authenticity score, which is the key to judging the difference level of data, because the essential purpose of the target domain discriminator D T is to increase the distribution gap between the generated sample (target domain aligned trajectory data X ST ) and the real sample (target domain real trajectory data X T ). The calculation method of the authenticity score is:

[0072]

[0073] Wherein, represents the authenticity score matrix, W dt represents all learnable parameters in the target domain discriminator D T . The element value I i,j ∈[0,1] in I, and the closer the value is to 1, the more it is considered to be a real sample; on the contrary, the closer the value is to 0, the more it is considered to be a generated sample.

[0074] Finally, the authenticity score I ST of the target domain aligned trajectory data X ST and the authenticity score I T of the target domain real trajectory data X TT will be used as the independent variable of the objective function to participate in the calculation of the target domain generation loss

[0075] For the source domain discriminator D S , which is responsible for distinguishing the source domain aligned trajectory data X T→S generated by the source domain generator G TS from the source domain real trajectory data X S , the authenticity score I TS of the source domain aligned trajectory data X TS and the authenticity score I S of the source domain real trajectory data X SS are calculated, and the calculation results are used to calculate the source domain generation loss L The calculation method of the authenticity score is as follows:

[0076]

[0077] Wherein, W ds represents all learnable parameters in the source domain discriminator D S .

[0078] Step S13: Calculate the target domain generation loss by combining the authenticity score of the target domain aligned trajectory data and the authenticity score of the target domain real trajectory data, and calculate the source domain generation loss by combining the authenticity score of the source domain aligned trajectory data and the authenticity score of the source domain real trajectory data, to obtain an optimal generation model, that is, the input of step S13 is the source domain real trajectory data X and the target domain real trajectory data X , and the output is the optimal generation model G

[0079] S131: Source domain trajectory data X S and target domain trajectory data X T are respectively sampled from a large amount of source domain data sets and a small amount of target domain data sets.

[0080] S132: The data-level alignment module is constructed and trained for 100 to 500 iterations, preferably 200 iterations, and in each iteration, a first target function is calculated:

[0081]

[0082] Wherein, represents the total loss function of the data-level alignment module; and are the target domain generation loss and the source domain generation loss, respectively, which are used to reduce the difference between the two domains; is a cycle consistency loss, which is used to ensure that the alignment process is stable and reliable; is an ontology loss, which is used to prevent the target domain generator G S→TAny operation is performed on the target domain data to improve the conversion efficiency. Among them, a and β are weight coefficients of the cycle consistency loss and the ontology loss, used to balance each loss so that their orders of magnitude are roughly the same. In the embodiment, a and β are both set to [8, 12], preferably, a and β are set to 10.

[0083] Step S133: The gradient of the neural network is calculated and the parameters of the source domain to target domain generator, the target domain to source domain generator, the source domain discriminator and the target domain discriminator are updated respectively, and the final optimization target is:

[0084]

[0085] The optimal generation model is obtained Wherein, D S represents the source domain discriminator. Then the target domain alignment trajectory data is generated The target domain alignment trajectory data S is mixed with the source domain real trajectory data X to obtain the enhanced source domain data.

[0086] Step S13 further includes dividing the enhanced source domain data by time length to obtain the source domain historical observation and the source domain future true value Similarly, the target domain data is divided to obtain the target domain historical observation and the target domain future true value Preparation for feature-level alignment.

[0087] Step S14: Establish a feature-level alignment embedding discriminator to map the enhanced source domain historical observation data and the target domain historical observation data to the same high-dimensional feature space, and generate the enhanced source domain historical observation data intermediate feature h S and the target domain historical observation data intermediate feature h T participating in domain alignment, respectively. The feature-level alignment embedding discriminator further calculates the authenticity scores of the enhanced source domain historical observation data intermediate feature and the target domain historical observation data intermediate feature.

[0088] Specifically, the feature-level alignment module uses the enhanced source domain historical observation data X' S and the target domain historical observation data X' T as input to further reduce the difference between the two domains at the feature level. Among them, the encoder Enc is responsible for mapping the trajectory data of the source domain and the target domain to the same high-dimensional feature space and generating the intermediate feature:

[0089]

[0090] Among them, denote two intermediate features participating in domain alignment align is the feature dimension size of both enc denote the learnable parameters in the encoder Enc. During the training process, the weights of the source domain encoder and the target domain encoder are shared and participate in optimization together.

[0091] Feature-level alignment embedding discriminator D E The task of D S is to output the true and false scores of h T and h E , and to maximize the gap between them as much as possible. The specific network structure of D Figure 5 is shown in the following figure. A simple multi-layer perceptron (MLP) network is used to build D E , and the output tensor of the nth layer is:

[0092]

[0093] Among them, a batch normalization (BN) layer is added after the first two layers of MLP to ensure the stability of the distribution of the score vector in each training batch. The final output of the network is a pair of true and false score vectors:

[0094]

[0095] wherein, represents the true and false score matrix, and W e represents all the learnable parameters in the feature-level alignment embedding discriminator. Similar to the true and false score matrix in the data-level alignment module, the value of each element in the score vector is I i ∈[0,1], and the closer the value is to 1, the more it is considered to be a target domain feature; on the contrary, the closer the value is to 0, the more it is considered to be a source domain feature. This embodiment tries to make the score of I S tend to 1.

[0096] Step S15: Combine the true and false scores of the intermediate features in the source domain historical observation data and the true and false scores of the intermediate features in the target domain historical observation data. Specifically, the input of step S15 is the enhanced source domain historical observation data X' S , the target domain historical observation data X' T , the source domain future true value Y S , and the output is an optimal generative model containing an optimal encoder Enc * and an optimal decoder Dec * , the optimal encoder outputs domain-invariant features that are common to different domains, and the optimal decoder generates pedestrian prediction trajectories based on the domain-invariant features.

[0097] Step S15 also includes constructing and iteratively training the feature-level alignment module 50 to 200 times, preferably 100 times, in which the second objective function is calculated in each iteration:

[0098]

[0099] in, The total loss function representing the feature-level alignment module. The prediction loss representing the trajectory, Represents the feature-level aligned embedding discriminator D E The adversarial loss is γ, which is a weighting coefficient for the adversarial loss and is used to balance the magnitudes of the various losses. γ is set to [1.0, 1.2], and preferably, in this embodiment, γ is set to 1.

[0100] For each loss, the encoder and decoder in the existing basic prediction model always aim to minimize them in order to achieve domain adaptation and output high-precision trajectories; while the feature-level aligned embedding discriminator always aims to maximize them to form a resistance, enhance the encoder and decoder's ability to withstand interference from feature-level domain differences, and improve the overall prediction model's cross-domain performance.

[0101] The goal of this stage is to train an optimal encoder Enc. * It can effectively output domain-invariant features that are universal across different domains, thereby completing the feature-level alignment task from the source domain to the target domain; and an optimal decoder Dec * Trajectories with low prediction errors can be generated based on domain-invariant features. Therefore, the overall training objective of the feature alignment module can be expressed as:

[0102]

[0103] Ultimately, the obtained domain-invariant features Decoding the domain-invariant features yields a high-precision predicted trajectory.

[0104] Step S21: Sample target domain data X from a large target domain dataset (an unknown scenario in a real-world application). T The optimal generative model calculates intermediate features of the target domain data: h T =Enc * (X T Finally, predict future trajectories based on intermediate features: This enables it to be used for cross-scenario prediction tasks in autonomous mobile robots or self-driving cars.

[0105] The experimental results of the method in this embodiment are as follows.

[0106]

[0107]

[0108] where A, B, C, D, E represent eth (ETH Zurich), hotel (hotel), univ (University of Cyprus), zara1 (Zara store in Cyprus, layout 1), zara2 (Zara store in Cyprus, layout 2) respectively.

[0109] The ADE (average displacement error) and FDE (final displacement error) results of the present embodiment (the last row of Table 1 and Table 2), six traditional methods (the first six rows of Table 1 and Table 2) and two currently most advanced domain adaptation prediction methods (the seventh to eighth rows of Table 1 and Table 2) on 20 tasks are reported in Table 1 and Table 2. In the table, bold font represents the optimal result, and underlined font represents the suboptimal result. Each cross-domain task is represented in the form of "S2T". The units of ADE and FDE are "meters". The lower the value, the higher the prediction accuracy of the model, that is, the stronger the domain adaptation ability of the model on the corresponding cross-domain task.

[0110] 1. First, compared with those methods that are not selected as the base model (the first to sixth rows of Table 1 and Table 2), the present embodiment has achieved a more obvious leading advantage on each task. Compared with the best-performing model SGCN, the average prediction performance of the present embodiment exceeds 73.8% in the ADE index and 73.4% in the FDE index. This shows that the complete network structure of the present embodiment can effectively reduce the differences between different domains and complete the knowledge transfer between different domains. Second, compared with the base prediction model Trajectron++, the present embodiment has produced a huge accuracy gain on most tasks. Overall, it has improved by 45.8% in ADE and 44.5% in FDE. This shows that the double alignment domain adaptation measure of the present embodiment is a general solution to the problem of general encoder-decoder structure models when applied across domains. In addition, hotel, as a scene with the smallest average pedestrian moving speed, its particularity of pedestrian movement makes the "B2X" task more challenging. As can be seen from the fact that all previous models have performed poorly on the B2X task, there is a huge domain difference between the hotel scene and other scenes. Compared with previous work, the breakthrough of the present embodiment on these B2X tasks further proves the particular effectiveness of the method of the present embodiment in reducing the domain gap between scenes with large differences.

[0111] 2. Secondly, compared with the best model AML in the last two domain adaptation prediction methods, the average ADE and FDE of the embodiment on all tasks are improved by 38.5% and 42.5%. Unlike T-GNN (the average ADE and FDE are improved by 66.7% and 66.5% respectively), the embodiment is divided into two stages: first, the coarse-grained domain alignment is completed by using the cycle GAN framework at the data level; second, instead of using a static and single inter-domain distance to calculate the alignment loss, the embodiment uses an adversarial loss to dynamically align the feature level information to further correct the domain shift of the high-dimensional feature space. Compared with AML, the embodiment first selects the input data end with rich macroscopic characteristics of the trajectory domain, realizes the migration of part of the scene information, and enables the subsequent model to focus more on the learning of the target domain; second, an auxiliary adversarial module is embedded in the middle feature layer to complete the migration of microscopic characteristics, and at the same time realize the optimization of the model. This is fundamentally different from the introduction of an additional post-processing module at the output end by AML. Experimental results prove that the method of the embodiment has more advantages: the domain adaptation measures of the embodiment can make the basic prediction model have better domain generalization performance.

[0112] The experimental results of the method of the embodiment of the application are described as follows.

[0113] The following experimental studies the universality and portability of the method of the embodiment of the application to other basic models. By applying the data-level alignment module to some excellent basic models, the effectiveness of the data-level alignment module for each basic prediction model is further studied. It should be noted that these models are all encoding-decoding network structures, which are compatible with the domain adaptation scheme of the embodiment of the application. In the experiment, the data-level alignment module will be used as a callable component of the model input in the training stage, providing aligned trajectory data X ST As shown in Table 3, all 20 cross-domain tasks are evaluated, and the first four rows in Table 3 represent the average performance of the original basic model in the prior art, and the results after applying the data-level alignment module are listed in the fifth to eighth rows. In general, the model trained by the data-level alignment module has achieved great precision gain (up to 42.4% / 46.1% in average ADE / FDE). This result further proves the richness of the macroscopic features hidden in the data, the necessity of domain adaptation processing for trajectory prediction models, and the universal applicability of the scheme provided by the embodiment of the application to other basic prediction models.

[0114] Table 3

[0115]

[0116] Among them, DLA represents the data-level alignment module.

[0117] In summary, the embodiment of the present application has the following characteristics:

[0118] 1. To solve the performance degradation problem of pedestrian trajectory prediction model in cross-domain test, the method provided by the embodiment of the present application trains the model by using a large amount of source domain data and a small amount of target domain data, so as to enhance the adaptability of the model to different domains (scenes); and the prior art either ignores the cross-domain application problem of the model or designs a specific prediction model to complete the domain adaptation, and does not have universal applicability.

[0119] 2. To solve the macroscopic difference problem between scenes, the domain alignment method for pedestrian trajectory data level of different scenes provided by the embodiment of the present application completes effective knowledge transfer in the form of preprocessing at the data level, can convert the pedestrian trajectory domain of the source domain to the target domain, so that the feature alignment module learns more target domain information; and the prior art ignores the data level characteristics of different domain input ports, has no measures to reduce the difference, and the cross-domain adaptability of the trained model is very limited.

[0120] 3. To solve the microscopic difference problem between scenes, the domain alignment method for pedestrian trajectory feature level provided by the embodiment of the present application can further transfer the source domain knowledge to the target domain, improve the ability of the model to generate domain universal knowledge while aligning the feature difference, reduce the prediction error of the model in the target domain, and improve the prediction accuracy of the model; and the prior art can only complete feature level alignment to reduce the difference between the source domain and the target domain, and does not have the ability to simultaneously improve the cross-domain adaptability and prediction accuracy of the model.

[0121] 4. The adversarial learning is adopted to dynamically complete the knowledge transfer between the source domain and the target domain, wherein the data alignment module applies a graph attention mechanism, can self-learn to complete implicit pedestrian social interaction feature extraction, improve the authenticity of the generated results, and is suitable for alignment between pedestrian trajectory domains with multiple scene difference types and uneven characteristic distribution. And the prior art needs to artificially define the distance function between domains, and such a static and preset function is only suitable for pedestrian movement scenes with relatively single motion mode.

[0122] 5. A time convolution network is adopted to realize encoding and decoding of the input trajectory, directly aggregates information of adjacent time nodes through convolution operation, and can effectively avoid the error accumulation problem of the recurrent neural network.

[0123] 6. The embodiment of the present application adopts multiple target functions to jointly optimize the domain alignment process, wherein the cycle consistency loss makes the data alignment link more stable and reliable. And most of the prior art only uses the Euclidean distance between the prediction result and the real trajectory as the target function to optimize the model, and the training effect is very limited.

[0124] The above is further detailed description of the present application in combination with specific preferred embodiments, and cannot be deemed as limitation of the specific implementation of the present application to these descriptions. For those skilled in the art to which the present application belongs, without departing from the concept of the present application, a number of equivalent substitutions or obvious variations can be made, and the performance or use is the same, which should be deemed as falling within the protection scope of the present application.

Claims

1. A cross-scene adaptive pedestrian trajectory prediction method, characterized in that, The method comprises the following steps: S11: converting source domain real trajectory data into target domain aligned trajectory data and converting target domain real trajectory data into source domain aligned trajectory data; S12: calculating the authenticity scores of the target domain aligned trajectory data, target domain real trajectory data, source domain aligned trajectory data and source domain real trajectory data respectively; S13: combining the authenticity scores of the target domain aligned trajectory data and the authenticity scores of the target domain real trajectory data to calculate a target domain generation loss, and combining the authenticity scores of the source domain aligned trajectory data and the authenticity scores of the source domain real trajectory data to calculate a source domain generation loss, and iteratively updating to obtain an optimal generation model; S21: sampling target domain data from a target domain data set, and the optimal generation model generates a future predicted trajectory of a pedestrian according to the target domain data, thereby being used for a cross-scene prediction task of an autonomous mobile robot or an automatic driving vehicle; In step S13, the optimal generation model further generates source domain aligned trajectory data, and mixes the source domain aligned trajectory data and source domain real trajectory data to obtain enhanced source domain data; The enhanced source domain data is divided into enhanced source domain historical observation data and source domain future true value data, and the target domain real trajectory data is divided into target domain historical observation data and target domain future true value data; The method further comprises the following steps: S14: mapping the enhanced source domain historical observation data and the target domain historical observation data to the same high-dimensional feature space, and respectively generating enhanced source domain historical observation data intermediate features and target domain historical observation data intermediate features participating in domain alignment, and then calculating authenticity scores of the enhanced source domain historical observation data intermediate features and authenticity scores of the target domain historical observation data intermediate features; S15: training an optimal generation model comprising an optimal encoder and an optimal decoder by combining the authenticity scores of the enhanced source domain historical observation data intermediate features and the authenticity scores of the target domain historical observation data intermediate features, wherein the optimal encoder outputs domain-invariant features that are common in different domains, and the optimal decoder generates a pedestrian predicted trajectory according to the domain-invariant features; In step S14, a feature-level alignment embedding discriminator is constructed, which outputs authenticity scores of the source domain historical observation data intermediate features and authenticity scores of the target domain historical observation data intermediate features according to the enhanced source domain historical observation data intermediate features and the target domain historical observation data intermediate features; the network structure of the feature-level alignment embedding discriminator is at least three layers, and the output of the nth layer is: wherein, r (n) represents the output tensor of the nth layer, the first two layers of the network structure are both followed by a batch normalization layer, and the network structure finally outputs the authenticity score I of the intermediate feature in the source domain historical observation data S and the authenticity score I of the intermediate feature in the target domain historical observation data T : wherein D E representing the feature-level alignment embedding discriminator, W e representing all learnable parameters in the feature-level alignment embedding discriminator, h S representing the intermediate feature in the enhanced source domain historical observation data, h T representing the intermediate feature in the target domain historical observation data.

2. The cross-scene adaptive pedestrian trajectory prediction method of claim 1, wherein, In step S11, a target domain generator and a source domain generator are respectively established, the target domain generator converts the source domain real trajectory data into the target domain aligned trajectory data, the input of the target domain generator is source domain real trajectory data, and the source domain generator converts target domain real trajectory data into source domain aligned trajectory data. The target domain generator first performs continuous time convolution on the time dimension through a time convolution network to obtain an intermediate tensor of the source domain real trajectory data; the time convolution network has a multi-head attention layer for outputting an expression tensor processed by an attention mechanism; the expression tensor is spliced with a feature dimension, and a full connection layer is used to adjust the dimension of the feature dimension to the same dimension as the time dimension.

3. The cross-scene adaptive pedestrian trajectory prediction method of claim 2, wherein, In step S12, a source domain discriminator and a target domain discriminator are respectively established, and the authenticity scores of the target domain aligned trajectory data and the target domain real trajectory data are derived by the target domain discriminator; The source domain discriminator derives the authenticity scores of the source domain aligned trajectory data and the source domain real trajectory data; The authenticity scores of the target domain aligned trajectory data and the target domain real trajectory data are calculated as follows: where I ST represents the authenticity score of the target domain aligned trajectory data, I TT represents the authenticity score of the target domain real trajectory data, X ST represents the target domain aligned trajectory data, X T represents the target domain real trajectory data, D T represents the target domain discriminator, W dt represents all the learnable parameters in the target domain discriminator.

4. The cross-scene adaptive pedestrian trajectory prediction method of claim 3, wherein, In step S13, the following steps are included: S131: source domain real trajectory data and target domain real trajectory data are respectively sampled from the source domain data set and the target domain data set; S132: a data-level alignment module is constructed and iteratively trained, and a first target function is calculated each time: wherein, a total loss function representing the data level alignment module, X S a source domain real trajectory data, a target domain generation loss, a source domain generation loss, a cycle consistency loss, a ontology loss, a is a weight coefficient of, and β is a weight coefficient of. S133: the neural network back propagation gradient is calculated, and the parameters of the target domain generator, the source domain generator, the source domain discriminator and the target domain discriminator are updated respectively to obtain an optimal generation model, and the expression of the optimal generation model is as follows: wherein G S→T represents a target domain generator, G T→S represents a source domain generator, D S represents a source domain discriminator, D T represents a target domain discriminator.

5. The cross-scenario adaptive pedestrian trajectory prediction method of claim 1, wherein, Step S15 includes: constructing and iteratively training a feature-level alignment module, and calculating a second target function each time: wherein, a total loss function representing the feature-level alignment module, a prediction loss representing the trajectory, an adversarial loss representing the discriminator, and γ is a weight coefficient of the adversarial loss; The input of the feature-level alignment module is the enhanced source domain historical observation data, the target domain historical observation data and the source domain future true value, and the output is the optimal encoder and the optimal decoder, and the training target of the feature-level alignment module is represented as: where Enc * denotes an optimal encoder, Dec * denotes an optimal decoder.

6. The cross-scene adaptive pedestrian trajectory prediction method of claim 5, wherein, Step S21, according to the target domain data, the intermediate features of the target domain data are calculated, and the future prediction trajectory of the pedestrian is predicted according to the intermediate features of the target domain data.

7. A cross-scenario adaptive pedestrian trajectory prediction device, comprising: The processor executes the computer program to implement the cross-scene adaptive pedestrian trajectory prediction method according to any one of claims 1-6. The processor executes the computer program to implement the cross-scene adaptive pedestrian trajectory prediction method according to any one of claims 1-6.

Images