Pedestrian trajectory prediction method based on hierarchical envelope domain adaptation

By constructing the MESC-HEDA model, multi-level envelope samples are generated and hierarchical envelope domain adaptation is solved, and the prediction accuracy and cross-domain adaptability are improved.

CN118918140AActive Publication Date: 2024-11-08CHONGQING UNIV +1

Patent Information

Application Number
CN202410942235.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-08
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

When facing the domain offset problem between different scenarios, the existing pedestrian trajectory prediction model has poor prediction accuracy, and the traditional method only considers local social hierarchical samples and local hierarchical domain adaptation, and fails to effectively deal with pedestrians and pedestrians and environment. multi-level complexity and uncertainty between them.

Method used

A pedestrian trajectory prediction method based on hierarchical envelope domain adaptation is proposed. By constructing a MESC-HEDA model, multi-level envelope samples are generated, including local hierarchical and individual hierarchical envelope samples, and domain distribution alignment is carried out in combination with attention mechanism to achieve cross-border Domain-time and spatial feature transfer learning.

Benefits of technology

The accuracy of pedestrian trajectory prediction and the multi-level cross-domain adaptability of the model are improved, and the trajectory data distribution differences between different scenarios are effectively dealt with, which enhances the transfer learning ability of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918140A_ABST
    Figure CN118918140A_ABST
Patent Text Reader

Abstract

The invention discloses a pedestrian trajectory prediction method based on hierarchical envelope domain adaptation. The method comprises the following steps: acquiring a source domain original trajectory sample and a target domain original trajectory sample; constructing a pedestrian trajectory prediction model based on hierarchical envelope domain adaptation, namely an MESC-HEDA model; the source domain original trajectory sample and the target domain original trajectory sample are utilized to train an MESC-HEDA model, pedestrian trajectories in the target domain original trajectory sample are predicted, the MESC-HEDA model comprises a two-stage envelope sample construction module, a global level envelope sample construction module and a hierarchical envelope domain adaptation module, and the global level envelope sample construction module is used for constructing a global level envelope sample. The two-stage envelope sample construction module comprises a local hierarchical envelope sample construction sub-module, an individual hierarchical envelope sample construction sub-module and a two-stage envelope sample feature fusion sub-module; the hierarchical envelope domain adaptation module comprises a local hierarchical envelope domain adaptation sub-module and a global hierarchical envelope domain adaptation sub-module. According to the invention, the problem of domain offset in the pedestrian trajectory scene is effectively solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of trajectory prediction, and in particular relates to a pedestrian trajectory prediction method based on hierarchical envelope domain adaptation. Background Art

[0002] Pedestrian trajectory prediction in complex environments is extremely challenging and is affected by many complex factors, including pedestrian movement status, destination, age, gender and other attributes of the pedestrians, as well as the interaction between multiple pedestrians. Most of the early research results were based on knowledge models such as social force models and constant velocity models, but the nonlinearity of pedestrian interaction and multiple complex factors lead to poor universality of the above methods and great limitations. In recent years, pedestrian trajectory prediction models based on machine learning, especially deep learning, have become a research hotspot, and the prediction accuracy has been improved compared with traditional algorithms. However, most methods based on deep learning models often assume that the marginal probability distribution of the training set and the test set trajectories is consistent, focusing on learning a unified interaction pattern between pedestrians and between pedestrians and the geographical environment, while ignoring the domain shift problem of the training set and the test set, resulting in poor model generalization performance. For example, in crowded places such as amusement parks or shopping malls, pedestrians interact frequently with each other, and usually need to constantly avoid, detour or wait to avoid collisions and obstructions, resulting in tortuous and complex trajectories, and constant mutations; while on the street, there are fewer collisions and interactions between pedestrians, and pedestrians tend to follow the direction of road planning and design, and their movement trajectories are mostly straight lines. Ignoring the above distribution differences between scenarios will inevitably affect the accuracy of pedestrian trajectory prediction when the model built from one domain is applied to another domain.

[0003] The performance analysis of pedestrian trajectory prediction models is mostly based on five trajectory scenarios: ETH, HOTEL, UNIV, ZARA1, and ZARA2. Quantitative evaluation of indicators such as sample number, number of pedestrians, walking speed, and acceleration found that there are significant differences between domains. The size of the same indicator in different scenarios is very different, which is usually called the pedestrian trajectory scenario domain offset problem. The usual solution is to regard the training scenario as the source domain, the test scenario as the target domain, and achieve distribution alignment through domain adaptation related methods. Although a pedestrian trajectory prediction model based on domain adaptation has emerged recently, it only considers local social level samples and only considers local level domain adaptation, which has limitations in the face of multi-level complexity and uncertainty of interactions between pedestrians and between pedestrians and the environment. In fact, the direction of pedestrian trajectories is jointly affected by three factors: the attributes of individual pedestrians, the social characteristics of neighboring pedestrians, and the characteristics of the global geographical environment. Therefore, it is necessary to have a pedestrian trajectory prediction method for domain adaptation. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a pedestrian trajectory prediction method based on hierarchical envelope domain adaptation.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A pedestrian trajectory prediction method based on hierarchical envelope domain adaptation, comprising:

[0007] Obtain source domain original trajectory samples and target domain original trajectory samples;

[0008] Construct a pedestrian trajectory prediction model based on hierarchical envelope domain adaptation, namely the MESC-HEDA model;

[0009] The MESC-HEDA model is trained using the original trajectory samples in the source domain and the original trajectory samples in the target domain, and the pedestrian trajectory in the original trajectory samples in the target domain is predicted.

[0010] The MESC-HEDA model includes: a two-level envelope sample construction module, a global level envelope sample construction module and a hierarchical envelope domain adaptation module, wherein the two-level envelope sample construction module includes a local level envelope sample construction submodule, an individual level envelope sample construction submodule, and a two-level envelope sample feature fusion submodule; the hierarchical envelope domain adaptation module includes a local level envelope domain adaptation submodule and a global level envelope domain adaptation submodule.

[0011] The local level envelope sample construction submodule is used to generate local level envelope samples according to the pedestrian adjacency relationship at the sample space level, including local level envelope samples of the source domain and local level envelope samples of the target domain.

[0012] The individual level envelope sample construction submodule is used to generate individual level envelope samples according to the pedestrian relationship at the individual level in the sample, including individual level envelope samples of the source domain and individual level envelope samples of the target domain.

[0013] The two-level envelope sample feature fusion submodule is used to generate two-level envelope samples according to the local level envelope samples and the individual level envelope samples, including the bipolar envelope samples of the source domain and the bipolar envelope samples of the target domain.

[0014] The global level envelope sample construction module is used to generate global level envelope samples according to the spatiotemporal feature distribution of the two-level envelope samples, including global level envelope samples of the source domain and global level envelope samples of the target domain.

[0015] The local level envelope domain adaptation submodule is used to determine the local level domain distribution alignment loss using the attention mechanism based on the local level envelope samples of the source domain and the target domain.

[0016] The global hierarchical envelope domain adaptation submodule is used to determine the global hierarchical domain distribution alignment loss based on the global hierarchical envelope samples of the source domain and the target domain.

[0017] Furthermore, according to the pedestrian adjacency relationship at the sample space level, a local level envelope sample is generated, including:

[0018] For any time t of all observed trajectories, that is, t∈{T1,...,T obs}, T1, T2, ..., T obs Indicates the moment of observing the trajectory, execute S21-S27:

[0019] S21: Get the original trajectory sequence samples of N pedestrians at time t, that is, the coordinates of N pedestrians at time t tr t i represents the coordinates of pedestrian i at time t, i∈{1, 2, ..., N};

[0020] S22: Processed as a graph representation G loc =(V loc , E loc ),in, Represents G loc The vertex set of Represents G loc The edge set of The edge formed by the coordinates of pedestrian i and pedestrian j is represented by the local level pedestrian adjacency relationship R loc and the local level adjacency matrix A loc express;

[0021] S23: Determine the feature vector F of N pedestrians at time t loc ,in, represents the velocity vector of pedestrian i at time t; and They represent the x and y components of the coordinates of pedestrian i at time t respectively; and They represent the x and y components of the coordinates of pedestrian i at time t-1 respectively;

[0022] S24: Mapping function F through local level envelope lhe , determine the local hierarchical pedestrian adjacency relationship R between any two pedestrians among all pedestrians loc , in, represents the speed of pedestrian j at time t, tr t j They represent the coordinates of pedestrian j at time t,

[0023]

[0024] and represents the x and y components of the coordinates of pedestrian j at time t; represents the Euclidean distance between pedestrians i and j after screening at time t; r1 and r2 represent the two set distances;

[0025] represents the velocity vector of pedestrian i at time t The angle between the line connecting the current position of pedestrian j and the angle between Taking the cosine function we get

[0026] It represents the comprehensive measure of the velocity angle between pedestrian i and pedestrian j at time t;

[0027]

[0028] S25: By introducing a threshold hyperparameter η∈[0, 1], R loc Perform local level envelope sample interaction strength screening to generate a local level mask M loc ,in, is the indicator function, when R loc When ≥η, output 1, otherwise output 0;

[0029] S26: Element-wise multiplication operation on R loc Screening is performed and the unit matrix I is superimposed to generate the local hierarchical adjacency matrix A loc , that is, A loc =M loc ⊙R loc +I, ⊙ means element-by-element multiplication;

[0030] S27: Zero-Softmax method is used to calculate the local hierarchical adjacency matrix A loc Normalize to generate a normalized local hierarchical adjacency matrix A' loc , thus obtaining the local level envelope sample corresponding to time t [F loc , A′ loc ];

[0031] For all observed trajectories, the corresponding F loc and A' loc Stacking is performed to obtain the local level envelope sample [F Loc , A Loc ], where F Loc ={F1, F2, ..., F obs}, F t represents the F corresponding to time t loc , A Loc={A′1,A′2,...,A′ obs},A′ t Denotes A' corresponding to time t loc , t∈{T1,...,T obs}.

[0032] Furthermore, according to the pedestrian relationships at the individual level in the sample, an individual level envelope sample is generated, including:

[0033] For any pedestrian i among all pedestrians, that is, i∈{1, 2, ..., N}, where N represents the total number of pedestrians, execute S31-S37:

[0034] S31: Obtain all observation trajectory times T1, T2, ..., T obs Coordinates of insider i in, tr t i represents the coordinates of pedestrian i at time t, t∈{T1,...,T obs};

[0035] S32: Processed as a graph representation G ind =(V ind , E ind ),in, G ind The vertex set of It's G ind The edge set of By the individual level pedestrian adjacency relation R ind and the individual level adjacency matrix A ind express;

[0036] S33: Pedestrian i is in the time period from T1 to T obs The corresponding eigenvector F ind , represents the velocity vector of pedestrian i at time t;

[0037] S34: Using the self-attention mechanism, determine the individual level pedestrian adjacency relationship R ind ,in,

[0038]

[0039] W Q and W K is a learnable parameter; Q ind and K ind Represent the query and key of the self-attention mechanism respectively; Indicates K indTranspose; D represents the scale factor; F′ ind =(F ind +PE)W ind Represents a high-dimensional tensor with position encoding information; PE = {Fram e t |t=T1,...,T obs} represents a single pedestrian trajectory tr t i In time period T l to T obs Position encoding tensor within Frame t Indicates the time frame number at time t; W ind represents a learnable parameter;

[0040] S35: By setting the threshold hyperparameter ψ∈[0, 1] ind Perform interactive screening of individual level envelope sample strength to generate individual level mask M ind ,in, is the indicator function, when R ind When ≥ψ, output 1, otherwise output 0;

[0041] S36: Mask the individual level M ind And the pedestrian adjacency relation R at the individual level ind Fusion is performed to obtain the individual level adjacency matrix A ind , that is, A ind =M ind ⊙R ind , ⊙ represents element-by-element multiplication;

[0042] S37: Use the Zero-Softmax method to perform the individual level adjacency matrix A ind Normalize and generate the normalized individual level adjacency matrix A′ ind , thus obtaining the individual level envelope sample corresponding to pedestrian i [F ind , A′ ind ];

[0043] F corresponding to all pedestrians loc and A′ loc Stacking is performed to obtain individual level envelope samples [F Ind , A Ind ], where F Ind ={F1, F2, ..., F N}, F i represents F corresponding to pedestrian i ind , A Ind ={A′1,A′2,...,A′ N},A′ iDenotes A′ corresponding to pedestrian i ind , i∈{1, 2, ..., N}.

[0044] Further, a two-level envelope sample is generated according to the local level envelope sample and the individual level envelope sample, including:

[0045] Perform graph convolution operation on the graph structure of the local level envelope sample to obtain the local level feature representation F′ of the local level envelope sample Loc , where F′ Loc =σ(A Loc F Loc W Loc ), σ represents the activation function ReLU(); W Loc represents a learnable parameter;

[0046] Perform graph convolution operation on the graph structure of the individual level envelope sample to obtain the individual level feature representation F′ of the individual level envelope sample Ind , where F′ Ind =σ(A Ind F Ind W Ind ), σ represents the activation function ReLU(); W Ind represents a learnable parameter;

[0047] The local level feature is represented as F′ Loc and individual level feature representation F′ Ind Perform splicing and fusion to obtain the feature representation F of the bipolar envelope sample tef ,in, Indicates splicing and fusion according to feature dimensions; represents the set of real numbers; L obs represents the total number of all observation moments; D represents the characteristic dimension of local level envelope samples and individual level envelope samples.

[0048] Furthermore, according to the temporal and spatial characteristic distribution of the two-level envelope samples, a global hierarchical envelope sample is generated, including:

[0049] The characteristic representation F of the bipolar envelope sample tef Input a multi-layer temporal convolutional network TCN with l layers, and the output of the previous layer is used as the input of the next layer to obtain the predicted trajectory, where for the l-th layer TCN:

[0050]

[0051] in, represents the input of the lth layer TCN, W (l) is the learnable parameter of the l-th layer TCN, Represents the output of the lth layer, that is, the predicted output of the multi-layer TCN;

[0052] The output of layer l The five parameters F of the binary Gaussian distribution of the predicted trajectory coordinates are determined through the linear layer final ,Right now Among them, Linear() represents linear layer processing, t∈[T obs+1 , T obs+2 , ..., T pre ], T obs+1 , T obs+2 , ..., T pre represents the time frame of the predicted trajectory, It represents the mean value parameter of the trajectory of pedestrian i at time t in the time frame of the predicted trajectory, which includes the mean value of the trajectory in the x and y directions, that is, and represents the standard deviation component of the trajectory of pedestrian i at time t, which includes the trajectory standard deviation in both the x and y directions, that is, and represents the correlation coefficient of pedestrian i’s trajectory at time t;

[0053] Based on the mean parameter of each pedestrian's trajectory Calculate the linear speed of the pedestrian at time t Right now

[0054] The prediction time period T obs+1 to T pre The linear velocity forms the vector U i ,Right now Among them, L pre Represents the forecast period T obs+1 to T pre The total number of predicted trajectory moments;

[0055] All pedestrians’ U i Splice to get the global level envelope sample Right now

[0056] Furthermore, according to the local level envelope samples of the source domain and the target domain, the attention mechanism is used to determine the local level domain distribution alignment loss, including:

[0057] Obtain local hierarchical feature representation of local hierarchical envelope samples of source and target domains and Among them, N s and N t Respectively represent the number of pedestrians in the local level envelope samples of the source domain and the target domain;

[0058] The local hierarchical features of the source domain and the target domain are represented as F′ Loc(s) and F′ Loc(t) Decomposed according to pedestrians, that is

[0059] in,

[0060] For each local level envelope sample in the source domain,

[0061] Will Flatten to D c The size of the feature vector, where D c =L obs ×D,

[0062] Determine the interaction weights of different pedestrians in the local hierarchical envelope samples of the source domain Right now in, express The transpose of

[0063] Determine the local hierarchical envelope of the source domain and the sum of the interactive effects between the pedestrian relationships J (s) ,Right now

[0064] The J corresponding to all local level envelope samples in the source domain (s) Stacking is performed to obtain the local level pedestrian attention feature concentrated representation M of the source domain (s) ,Right now Among them, K (s) Represents the number of local hierarchical envelope samples in the source domain;

[0065] For each local level envelope sample in the target domain,

[0066] Will Flatten to D c The size of the feature vector, where D c =L obs ×D,

[0067] Determine the interaction weights of different pedestrians in the local hierarchical envelope sample of the target domain Right now in, express The transpose of

[0068] Determine the local hierarchical envelope of the target domain and the sum of the interactions between the samples J (t) ,Right now

[0069] The J corresponding to all local level envelope samples in the target domain(t) Stacking is performed to obtain the local level pedestrian attention feature concentrated representation M of the target domain (t) ,Right now Among them, K (t) Represents the number of local hierarchical envelope samples in the target domain;

[0070] The local hierarchical pedestrian attention features of the source domain and the target domain are centrally represented as M. (s) and M (t) , determine the local hierarchical domain distribution alignment loss L loc , that is, L loc = Dist1(M (s) , M (t) ), Dist1() represents the first distance function.

[0071] Furthermore, according to the global hierarchical envelope samples of the source domain and the target domain, the global hierarchical domain distribution alignment loss is determined, including:

[0072] Get the global hierarchical envelope samples F of the source domain and the target domain glo(s) and F glo(t) ;

[0073] Determine the linear velocity difference ratio between the source domain trajectory and the target domain trajectory represents the linear velocity of pedestrian i in the source domain at time t, represents the linear velocity of pedestrian j in the target domain at time t, N s represents the total number of pedestrians in the source domain in the training set, N t represents the total number of pedestrians in the target domain in the validation set;

[0074] Determine the global hierarchical domain distribution alignment loss L glo , that is, L glo = Dist2(F glo(s) ,κ×F glo(t) ), where Dist2() represents the second distance function.

[0075] Furthermore, the objective function L when training the MESC-HEDA model using the original trajectory samples in the source domain and the original trajectory samples in the target domain is:

[0076] L=L pre +λ1L loc +λ2L glo ,

[0077] Among them, L pre represents the prediction loss, λ1 and λ2 represent weights, N represents the total number of pedestrians in the source domain,

[0078]

[0079] represents the probability density function, It represents the mean value parameter of the trajectory of pedestrian i at time t in the time frame of the predicted trajectory, which includes the mean value of the trajectory in the x and y directions, that is, and represents the standard deviation component of the trajectory of pedestrian i at time t, which includes the trajectory standard deviation in both the x and y directions, that is, and t represents the correlation coefficient of the trajectory of pedestrian i at time t.

[0080] The beneficial effects of the present invention are:

[0081] The present invention designs individual level envelope samples and combines them with local level envelope samples to construct bipolar envelope samples. According to the spatiotemporal characteristics of the bipolar envelope samples, global envelope samples are further generated, thereby constructing multi-level envelope samples. This can obtain richer spatiotemporal feature representations of pedestrian trajectory data at different levels.

[0082] The present invention designs a local hierarchical domain adaptation module and a global hierarchical domain adaptation module between the source domain and the target domain for each level of envelope samples, thereby realizing hierarchical envelope domain adaptation. Transfer learning through the hierarchical envelope domain adaptation module can better adapt to the multi-level complexity and uncertainty between pedestrians and between pedestrians and the environment.

[0083] The present invention further enhances the model's multi-level cross-domain spatiotemporal feature transfer learning capability by weighted fusion of domain distribution differences represented by local-level and global-level features, thereby realizing pedestrian trajectory prediction under domain distribution alignment.

[0084] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0086] Figure 1 It is a flow chart of the pedestrian trajectory prediction method based on hierarchical envelope domain adaptation;

[0087] Figure 2 It is the block diagram of MESC-HEDA model;

[0088] Figure 3 It is a schematic diagram of the construction module of the local level envelope sample;

[0089] Figure 4 It is a schematic diagram of the construction module of the individual level envelope sample. DETAILED DESCRIPTION

[0090] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments are only for illustrating the present invention, rather than for limiting the protection scope of the present invention.

[0091] The present invention proposes a pedestrian trajectory prediction model (MESC-HEDA) based on hierarchical envelope domain adaptation (HEDA). MESC stands for Multi-level Envelope Sample Construction. First, by constructing the local level pedestrian adjacency relationship, the local level envelope sample construction module is designed, and through the individual level pedestrian relationship, the individual level envelope sample construction module is designed, and the two are fused to form a two-stage envelope sample construction (TESC) module; then, based on the two-stage envelope sample construction module, the spatiotemporal feature distribution of all pedestrian trajectories is calculated to construct a global level envelope sample; then, based on the attention mechanism and cross-domain distribution alignment, the local level envelope domain adaptation (LEDA) module and the global level envelope domain adaptation (GEDA) module are designed respectively, and then a weighted prediction loss function is constructed to integrate the two and perform joint optimization. The model has low error and is stable, and can be well qualified for cross-domain pedestrian trajectory prediction.

[0092] In the given scene of the present invention, N pedestrians are in the time period from T1 to T obs Observation trajectory The learnable parameter is W * The model f(·) (i.e., MESC-HEDA model) can predict the future short time period T obs+1 to T pre The trajectory in represents the two-dimensional spatial coordinates of the i-th pedestrian at time frame t, expressed as follows:

[0093]

[0094] Figure 1This is a flow chart of the pedestrian trajectory prediction method based on hierarchical envelope domain adaptation. Figure 1 As shown, the method includes:

[0095] Obtain source domain original trajectory samples and target domain original trajectory samples;

[0096] Obtain a pedestrian trajectory prediction model based on hierarchical envelope domain adaptation, namely the MESC-HEDA model;

[0097] The MESC-HEDA model is trained using the original trajectory samples in the source domain and the original trajectory samples in the target domain, and the pedestrian trajectories in the original trajectory samples in the target domain are predicted.

[0098] Figure 2 This is the block diagram of the MESC-HEDA model. Figure 2 As shown, the MESC-HEDA model includes: a two-level envelope sample construction module, a global level envelope sample construction module and a hierarchical envelope domain adaptation module, wherein the two-level envelope sample construction module includes a local level envelope sample construction submodule (also referred to as "local level envelope sample construction module"), an individual level envelope sample construction submodule (also referred to as "individual level envelope sample construction module"), and a two-level envelope sample feature fusion submodule (also referred to as "two-level envelope sample feature fusion module"); the hierarchical envelope domain adaptation module includes a local level envelope domain adaptation submodule (also referred to as "local level envelope domain adaptation module") and a global level envelope domain adaptation submodule (also referred to as "global level envelope domain adaptation module").

[0099] The local level envelope sample construction submodule is used to generate local level envelope samples according to the pedestrian adjacency relationship at the sample space level, including local level envelope samples of the source domain and local level envelope samples of the target domain. That is, when the local level envelope sample construction submodule processes samples of the source domain, local level envelope samples of the source domain are generated; when the local level envelope sample construction submodule processes samples of the target domain, local level envelope samples of the source domain are generated.

[0100] The individual level envelope sample construction submodule is used to generate individual level envelope samples according to the pedestrian relationships at the individual level in the samples, including individual level envelope samples of the source domain and individual level envelope samples of the target domain. That is, when the individual level envelope sample construction submodule processes the samples of the source domain, the individual level envelope samples of the source domain are generated; when the individual level envelope sample construction submodule processes the samples of the target domain, the individual level envelope samples of the source domain are generated.

[0101] The two-level envelope sample feature fusion module is used to generate two-level envelope samples according to the local level envelope samples and the individual level envelope samples, including bipolar envelope samples of the source domain and bipolar envelope samples of the target domain, that is, when the two-level envelope sample feature fusion module processes the local level envelope samples and the individual level envelope samples of the source domain, the bipolar envelope samples of the source domain are generated; when the two-level envelope sample feature fusion module processes the local level envelope samples and the individual level envelope samples of the target domain, the bipolar envelope samples of the target domain are generated.

[0102] The global hierarchical envelope sample construction module is used to generate global hierarchical envelope samples according to the spatiotemporal feature distribution of the bipolar envelope samples, including global hierarchical envelope samples of the source domain and global hierarchical envelope samples of the target domain. That is, when the global hierarchical envelope sample construction module processes the bipolar envelope samples of the source domain, the global hierarchical envelope samples of the source domain are generated; when the global hierarchical envelope sample construction module processes the bipolar envelope samples of the target domain, the global hierarchical envelope samples of the target domain are generated.

[0103] The local level envelope domain adaptation submodule is used to determine the local level domain distribution alignment loss using the attention mechanism based on the local level envelope samples of the source domain and the target domain.

[0104] The global hierarchical envelope domain adaptation submodule is used to determine the global hierarchical domain distribution alignment loss based on the global hierarchical envelope samples of the source domain and the target domain.

[0105] The existing deep learning concept is mainly based on the modeling of raw data samples, automatically extracting the deep features that reflect the raw data samples, while the envelope sample learning mechanism proposed in this invention aims to mine the correlation information between the raw samples and transform them into new samples, so that the raw samples can be mapped into multi-layer envelope samples, which can characterize the target object from multiple levels. Studies have shown that compared with the existing modeling method directly based on the original samples, the modeling method based on hierarchical envelope samples helps to improve the accuracy. The envelope sample learning mechanism is briefly described as follows: Given a data set X = [x1, x2, ..., x n ] T , for any sample x i ∈X, find its nearest neighbor samples through operator η1 and form a nearest neighbor sample set N(x i ), Then x i and N(x i ) into a new sample set X c =[x i ,N(x i )] T , and construct the adjacency matrix A(X c ) expresses its adjacency relationship, and obtains the sample set [X c , A(X c)] is the process of sample envelopment; secondly, through the transformation operator ψ1, the sample set [X c , A(X c )] to transform X e =ψ1([X c , A(X c )]), then the generated new sample set X e is the envelope sample. Transform from X to X e The process is called envelope sample transformation. Multi-level envelope sample set And the existing sample set X constitutes a hierarchical envelope sample space in Based on this, the model H(X) based on the original sample is transformed into a model based on multi-level envelope samples That is, a multi-level envelope sample construction is realized, wherein L represents the total number of envelope sample transformation operations, and after L envelope sample transformation operations, L level envelope sample sets are obtained.

[0106] Figure 3 It is a schematic diagram of the local level envelope sample construction module. Figure 3 As shown in the figure, based on the original pedestrian trajectory sequence samples in the scene (which can be the source domain or the target domain), the relationship between each pedestrian in the spatial dimension is used to map the local hierarchical envelope mapping function F lhe The adjacency relationship of the original trajectory samples at the spatial level is mined to construct a series of spatial graph structures with adjacency matrices; then the local level envelope samples are obtained through appropriate interaction strength screening and normalization operations [F loc , A′ loc ] to realize envelope sample transformation.

[0107] Specifically, in combination with the description in the following "Module 1", the local level envelope sample construction module generates local level envelope samples according to the pedestrian adjacency relationship at the sample space level, including:

[0108] For any time t of all observed trajectories, that is, t∈{T1,...,T obs}, T1, T2, ..., T obs Indicates the moment of observing the trajectory, execute S21-S27:

[0109] S21: Obtain the original trajectory sequence samples of N pedestrians at time t (i.e., the trajectory data of each pedestrian at the same time in the same scene), i.e., the coordinates of N pedestrians at time t tr t i represents the coordinates of pedestrian i at time t, i∈{1, 2, ..., N};

[0110] S22: Processed as a graph representation G loc =(V loc , E loc ),in, Represents G loc The vertex set of Represents G loc The edge set of represents the edge formed by the coordinates of pedestrian i and pedestrian j, where each edge is composed of the local level pedestrian adjacency relation R loc and the local level adjacency matrix A loc express;

[0111] S23: Determine the feature vector F of N pedestrians at time t loc ,in, represents the velocity vector of pedestrian i at time t; and They represent the x and y components of the coordinates of pedestrian i at time t respectively; and They represent the x and y components of the coordinates of pedestrian i at time t-1 respectively;

[0112] S24: Mapping function F through local level envelope lhe , determine the local hierarchical pedestrian adjacency relationship R between any two pedestrians among all pedestrians loc , R loc Taking into account the local hierarchical interaction relationship of pedestrian groups (i.e., the relationship between all pedestrians in the same scene at the same time), two local hierarchical pedestrian group envelopes with radii r1 and r2 are defined. When the distance between other pedestrians and the target pedestrian is within the range of the two circles, these pedestrians can be regarded as local hierarchical envelope samples. represents the Euclidean distance between pedestrians i and j after screening at time t, that is,

[0113]

[0114] and represents the x and y components of the coordinates of pedestrian j at time t; represents the Euclidean distance between pedestrians i and j after screening at time t; r1 and r2 represent the two set distances; represents the velocity vector of pedestrian i at time t The angle between the line connecting the current position of pedestrian j and the angle between Taking the cosine function we get

[0115]

[0116] It represents the comprehensive measure of the velocity angle between pedestrian i and pedestrian j at time t;

[0117] definition, In summary, the local level envelope mapping function F lhe () includes many factors such as the speed of pedestrians in the group and the distance between the surrounding pedestrians, as well as the angle in the relative motion direction. The local level pedestrian adjacency relationship R loc The construction is as follows:

[0118]

[0119] in, represents the speed of pedestrian j at time t, tr t j They represent the coordinates of pedestrian j at time t,

[0120] S25: By introducing a threshold hyperparameter η∈[0, 1], R loc Perform local level envelope sample interaction strength screening to generate a local level mask M loc ,Right now

[0121]

[0122] in, is the indicator function, when R loc When ≥η, output 1, otherwise output 0;

[0123] S26: Element-wise multiplication operation on R loc Screening is performed and the unit matrix I is superimposed to generate the local hierarchical adjacency matrix A loc ,Right now:

[0124] A loc =M loc ⊙R loc +I, (6)

[0125] Among them, ⊙ represents element-by-element multiplication;

[0126] S27: Zero-Softmax method is used to calculate the local hierarchical adjacency matrix A loc Normalize to generate a normalized local hierarchical adjacency matrix A′ loc , thus obtaining the local level envelope sample corresponding to time t [F loc , A′ loc ];

[0127] For all observed trajectories, the corresponding F loc and A′ loc Stacking is performed to obtain the local level envelope sample [Floc , A Loc ], where F Loc ={F1, F2, ..., F obs}, F t represents the F corresponding to time t loc , A Loc ={A′1,A′2,...,A′ obs},A′ t Denotes A′ corresponding to time t loc , t∈{T1,...,T obs}.

[0128] In step S27, the local level adjacency matrix A loc Each element in Normalization can be expressed as:

[0129]

[0130] where ∈ is a very small constant to ensure numerical stability.

[0131]

[0132]

[0133] Figure 4 is a schematic diagram of the individual level envelope sample construction module. Figure 4 As shown in Figure 1, based on the original pedestrian trajectory sequence samples in the scene (which can be the source domain or the target domain), the association relationship of the same pedestrian in the time dimension is used to mine the adjacency relationship of the original trajectory samples at the time level, thereby constructing a series of time graph structures with adjacency matrices; then, through appropriate interaction strength screening and normalization operations, the individual level envelope samples [F ind , A′ ind ] to realize envelope sample transformation.

[0134] Specifically, in combination with the description in the following "Module 2", the individual level envelope sample construction module generates individual level envelope samples according to the pedestrian relationship at the individual level in the sample, including:

[0135] For any pedestrian i among all pedestrians, that is, i∈{1, 2, ..., N}, where N represents the total number of pedestrians, execute S31-S37:

[0136] S31: Obtain all observation trajectory times T1, T2, ..., T obs Coordinates of insider i in, tr t irepresents the coordinates of pedestrian i at time t, t∈{T1,...,T obs};

[0137] S32: Processed as a graph representation G ind =(V ind , E ind ),in, G ind The vertex set of It's G ind The edge set of By the individual level pedestrian adjacency relation R ind and the individual level adjacency matrix A ind express;

[0138] S33: Pedestrian i is in the time period from T1 to T obs The corresponding eigenvector F ind , represents the velocity vector of pedestrian i at time t;

[0139] S34: Using the self-attention mechanism, determine the individual level pedestrian adjacency relationship R ind , Frame t Indicates the number of time frames at time t. Assuming that data is collected every 0.4s, the number of time frames corresponding to time t is equivalent to: 0.4s (i.e. T1) corresponds to time frame 1, 0.8s (i.e. T2) corresponds to time frame 2, 1.2s (T3) corresponds to time frame 3, and so on. That is, T1 is 0.4s, T2 is 0.8s, T3 is 1.2s, until T obs (Assuming there are 8 sampling moments) is 3.2s, and Frame t is the frame number, which is an integer, 1, 2, ..., 8.

[0140] PE = {Frame t |t=T1,...,T obs} represents a single pedestrian trajectory tr t i In the time period T1 to T obs The position encoding tensor in the original individual level feature vector F ind Based on this, we add a position encoding tensor PE, which can be learned through the parameter W ind Perform a linear transformation to obtain a high-dimensional tensor F′ with position encoding information ind ,Right now:

[0141] F′ ind =(F ind +PE)W ind , (8)

[0142] Through the learnable parameter W Q and W K The query and key of the self-attention mechanism are obtained respectively. Finally, the product of query and key is scaled by the scaling factor D to maintain stability and normalized by the Softmax function to obtain the individual level envelope sample pedestrian adjacency relationship R ind

[0143]

[0144] Among them, W Q and W K is a learnable parameter; Q ind and K ind Represent the query and key of the self-attention mechanism respectively; Indicates K ind Perform transposition;

[0145] S35: By setting the threshold hyperparameter ψ∈[0, 1] ind Perform interactive screening of individual level envelope sample strength to generate individual level mask M ind ,Right now:

[0146]

[0147] in, is the indicator function, when R ind When ≥ψ, output 1, otherwise output 0;

[0148] S36: Mask the individual level M ind And the pedestrian adjacency relation R at the individual level ind Fusion is performed to obtain the individual level adjacency matrix A ind ,Right now:

[0149] A ind =M ind ⊙R ind , (11)

[0150] Among them, ⊙ represents element-by-element multiplication;

[0151] S37: Use the Zero-Softmax method to perform the individual level adjacency matrix A ind Normalize and generate the normalized individual level adjacency matrix A′ ind , thus obtaining the individual level envelope sample corresponding to pedestrian i [F ind , A′ ind ];

[0152] F corresponding to all pedestrians loc and A′ locStacking is performed to obtain individual level envelope samples [F Ind , A Ind ], where F Ind ={F1, F2, ..., F N}, F i represents F corresponding to pedestrian i ind , A Ind ={A′1,A′2,...,A′ N},A′ i Denotes A′ corresponding to pedestrian i ind , i∈{1, 2, ..., N}.

[0153] Among them, in S37, for the individual level adjacency matrix A ind After normalization, it can be expressed as:

[0154] A′ ind =Zero-Softmax(A ind ), (11)

[0155]

[0156] The bipolar envelope sample feature fusion module is described below.

[0157] Combined with the description in the following "Module 3", the bipolar envelope sample feature fusion module generates a bipolar envelope sample according to the local level envelope sample and the individual level envelope sample, including:

[0158] Perform graph convolution operation on the graph structure of the local level envelope sample to obtain the local level feature representation F′ of the local level envelope sample Loc ,Right now:

[0159] F′ Loc =σ(A Loc F Loc W Loc ), (12)

[0160] Where σ represents the activation function ReLU(); W Loc represents a learnable parameter;

[0161] Perform graph convolution operation on the graph structure of the individual level envelope sample to obtain the individual level feature representation F′ of the individual level envelope sample Ind ,Right now:

[0162] F′ Ind =σ(A Ind F Ind W Ind ), (13)

[0163] Where σ represents the activation function ReLU(); W Ind represents a learnable parameter;

[0164] The local level feature is represented as F′ Loc and individual level feature representation F′ Ind Perform splicing and fusion to obtain the feature representation F of the bipolar envelope sample tef ,Right now:

[0165]

[0166] in, Indicates splicing and fusion according to feature dimensions; represents the set of real numbers; L obs represents the total number of all observation moments; D represents the characteristic dimension of local level envelope samples and individual level envelope samples.

[0167]

[0168] The global level envelope sample construction module is described below.

[0169] The global level envelope sample construction module generates global level envelope samples according to the temporal and spatial characteristic distribution of the two-level envelope samples, including:

[0170] The characteristic representation F of the bipolar envelope sample tef Input a multi-layer temporal convolutional network (TCN) with l layers, and the output of the previous layer is used as the input of the next layer to obtain the predicted trajectory. For the l-th layer of TCN:

[0171]

[0172] in, represents the input of the lth layer TCN, W (l) is the learnable parameter of the l-th layer TCN, Represents the output of the lth layer, that is, the predicted output of a multi-layer TCN; the number of layers of TCN can be arbitrary, for example, 3 layers, 4 layers, 5 layers, etc.;

[0173] The output of the lth layer (i.e. the last layer) The five parameters F of the binary Gaussian distribution of the predicted trajectory coordinates are determined through the linear layer final ,Right now:

[0174]

[0175] Among them, Linear() represents linear layer processing, t∈[T obs+1 , T obs+2, ..., T pre ], T obs+1 , T obs+2 , ..., T pre Represents the time frame of the predicted trajectory. Through this linear layer, trajectory prediction can be performed for all prediction moments. It represents the mean value parameter of the trajectory of pedestrian i at time t in the time frame of the predicted trajectory, which includes the mean value of the trajectory in the x and y directions, that is, and represents the standard deviation component of the trajectory of pedestrian i at time t, which includes the trajectory standard deviation in both the x and y directions, that is, and represents the correlation coefficient of pedestrian i’s trajectory at time t;

[0176] Then the global level envelope sample is determined according to the following formula, namely:

[0177]

[0178] Specifically, based on the mean trajectory parameter of each pedestrian Calculate the linear velocity of the pedestrian at each time frame t Right now

[0179] The prediction time period T obs+1 to T pre The linear velocity forms the vector U i ,Right now Among them, L pre Represents the forecast period T obs+1 to T pre The total number of predicted trajectory moments;

[0180] All pedestrians’ U i Splice to get the global level envelope sample Right now

[0181] In order to solve the problem of inconsistent distribution of trajectory features in the source domain and the target domain, this paper constructs a multi-level hierarchical envelope domain adaptation module. What is significantly different from the existing domain adaptation methods is that a single "sample" in pedestrian trajectory prediction is composed of a combination of different numbers of pedestrian trajectories, that is, there are problems of different scales and dimensions. Extracting unified features from samples of different scales will result in the inability to finely characterize sample features, resulting in information loss. On the one hand, this paper adopts a sample feature extraction method based on the attention mechanism, which can more effectively adapt to different sample scales and avoid the problem of inaccurate local and global domain adaptation caused by feature scale representation errors. On the other hand, the combination of local and global envelope samples contains all pedestrian motion information, which can well solve the domain offset problem.

[0182] The present invention designs a hierarchical envelope domain adaptation module, which includes a local level envelope domain adaptation submodule and a global level envelope domain adaptation submodule. The function of the hierarchical envelope domain adaptation module is described below in conjunction with the description in "Module 4".

[0183] The local level envelope domain adaptation submodule can determine the local level domain distribution alignment loss using the attention mechanism based on the local level envelope samples of the source domain and the target domain, including:

[0184] Obtain local hierarchical feature representation of local hierarchical envelope samples of source and target domains and Among them, N s and N t Respectively represent the number of pedestrians in the local level envelope samples of the source domain and the target domain;

[0185] The local hierarchical features of the source domain and the target domain are represented as F′ Loc(s) and F′ Loc(t) Decomposed by pedestrians, namely:

[0186]

[0187] in,

[0188] For each local level envelope sample in the source domain,

[0189] Will Flatten to D c The size of the feature vector, where D c =L obs ×D,

[0190] Determine the interaction weights of different pedestrians in the local hierarchical envelope samples of the source domain Right now:

[0191]

[0192] in, express The transpose of the different pedestrian features in the same local level envelope sample The product of and expresses the weight of the interaction relationship, and calculates the cumulative sum of all the interactions between two pedestrians. The attention score is the proportion of two specific pedestrians in it. Obviously, the larger the proportion value, the stronger the interaction effect, otherwise the weaker. Feature number D c It plays a normalizing role.

[0193] Determine the local hierarchical envelope of the source domain and the sum of the interactive effects between the pedestrian relationships J (s) ,Right now:

[0194]

[0195] The J corresponding to all local level envelope samples in the source domain (s) Stacking is performed to obtain the local level pedestrian attention feature concentrated representation M of the source domain (s) ,Right now:

[0196]

[0197] Among them, K (s) Represents the number of local hierarchical envelope samples in the source domain;

[0198] For each local level envelope sample in the target domain,

[0199] Will Flatten to D c The size of the feature vector, where D c =L obs ×D,

[0200] Determine the interaction weights of different pedestrians in the local hierarchical envelope sample of the target domain Right now:

[0201]

[0202] in, express The transpose of the different pedestrian features in the same local level envelope sample The product of and expresses the weight of the interaction relationship, and calculates the cumulative sum of all the interactions between two pedestrians. The attention score is the proportion of two specific pedestrians in it. Obviously, the larger the proportion value, the stronger the interaction effect, otherwise the weaker. Feature number D c It plays a normalizing role.

[0203] Determine the local hierarchical envelope of the target domain and the sum of the interactions between the samples J (t) ,Right now:

[0204]

[0205] The J corresponding to all local level envelope samples in the target domain (t) Stacking is performed to obtain the local level pedestrian attention feature concentrated representation M of the target domain (t) ,Right now:

[0206]

[0207] Among them, K (t) Represents the number of local hierarchical envelope samples in the target domain;

[0208] The local hierarchical pedestrian attention features of the source domain and the target domain are centrally represented as M. (s) and M (t) , determine the local hierarchical domain distribution alignment loss L loc ,Right now:

[0209] L loc = Dist1(M (s) , M (t) ), (twenty two)

[0210] Wherein, Dist1() represents the first distance function.

[0211] The global hierarchical domain adaptation submodule determines the global hierarchical domain distribution alignment loss based on the global hierarchical envelope samples of the source domain and the target domain, including:

[0212] Get the global hierarchical envelope samples F of the source domain and the target domain glo(s) and F glo(t) ;

[0213] Determine the linear velocity difference ratio between the source domain trajectory and the target domain trajectory represents the linear velocity of pedestrian i in the source domain at time t, represents the linear velocity of pedestrian j in the target domain at time t, N s Nt represents the total number of pedestrians in the source domain in the training set, and Nt represents the total number of pedestrians in the target domain in the validation set;

[0214] Determine the global hierarchical domain distribution alignment loss L glo ,Right now:

[0215] L glo = Dist2(F glo(s) ,κ×F glo(t) ), (twenty three)

[0216] Wherein, Dist2() represents the second distance function.

[0217] There are many options for distance functions Dist1() and Dist2(), such as MMD loss, CORAL loss, GFK loss, and L2 distance, etc. The present invention further enhances the transfer learning capability of the network's multi-level cross-domain spatiotemporal features by jointly calculating the distribution differences of local hierarchical features and global hierarchical features of the target domain and the source domain.

[0218]

[0219]

[0220] Assume that the pedestrian coordinates It follows a bivariate Gaussian distribution Represents the mean, which includes the mean of the trajectories in the x and y directions, that is, and Represents the standard deviation, which includes the trajectory standard deviation in both the x and y directions, that is, and represents the correlation coefficient, and N represents the total number of pedestrians in the source domain. Therefore, the prediction loss L of the predicted trajectory pre It can be transformed into minimizing the negative log-likelihood loss:

[0221]

[0222] in, represents the probability density function.

[0223] The objective function L when training the MESC-HEDA model using the original trajectory samples in the source domain and the original trajectory samples in the target domain is:

[0224] L=L pre +λ1L loc +λ2L glo , (25)

[0225] Among them, λ1 and λ2 represent weights, which are used to balance the three losses and predict the loss L pre It is only related to the source domain samples and has nothing to do with the target domain samples. The model jointly minimizes the prediction loss L of the trajectory pre , the local hierarchical domain distribution alignment loss L between the source domain and the target domain loc and the global hierarchical domain distribution alignment loss L glo Conduct training.

[0226] The advantages of the present invention are illustrated below through specific simulation experiments.

[0227] Dataset: The experiment selected two representative public datasets in the real world: ETH and UCY. These two datasets have been widely used in a large number of studies on pedestrian trajectory prediction methods. The dataset contains social interactions between pedestrians and surrounding pedestrians and interactions between pedestrians and the environment, such as the gathering and staying of crowds, the avoidance of crowds and collisions, and group movement. The ETH dataset consists of two scenes, ETH and HOTEL, and the UCY dataset consists of three scenes, UNIV, ZARA1, and ZARA2. Among them, the UCY dataset scenes tend to be scenes with dense crowds, which often contain more complex pedestrian social interactions, while the ETH dataset scenes tend to be scenes with sparse crowds, which means that there are fewer social interactions between pedestrians.

[0228] Experimental setup: Referring to the recognized paradigm of existing trajectory prediction research based on domain adaptation, each scene is regarded as a trajectory domain. The model is first trained on one domain, then verified on the validation sets of the other four domains, and finally tested on the other four domains respectively, and the effectiveness of the model is verified by comparing its prediction accuracy. In the present invention, A, B, C, D, and E represent the pedestrian trajectory domains of the five scenes of ETH, HOTEL, UNIV, ZARA1, and ZARA2 respectively; there are a total of 20 trajectory prediction tasks: A→B / C / D / E, B→A / C / D / E, C→A / B / D / E, D→A / B / C / E, and E→A / B / C / D. During the training phase, the model of the present invention can only access the trajectories observed in the validation set, that is, the validation set is only used for domain adaptation.

[0229] Evaluation indicators: The model performance evaluation indicators use average displacement error (ADE) and final displacement error (FDE). ADE is used to evaluate the average prediction accuracy of the predicted trajectory, and FDE is used to evaluate the prediction accuracy of the endpoint of the last frame trajectory. Currently, since Social-LSTM, Social-STGCNN and Social-GAN all use the evaluation criterion that the model output obeys the bivariate Gaussian distribution, the present invention also follows this unified criterion for performance comparison.

[0230] Parameter setting: The experimental software platform is Pytorch=2.3, the hardware platform uses CPU i9-13900H GPU uses RTXA3000, and the memory is 64G. The observed pedestrian history trajectory is 3.2 seconds (8 time frames), and the predicted trajectory is 4.8 seconds (12 time frames). The number of GCN layers is 1, the number of TCN layers is 5, and the feature dimension is set to 16. In the training stage, the batch size is set to 16, the local level domain distribution alignment loss weight is set to 9, the global level domain distribution alignment loss weight is set to 5, the local level envelope sample threshold screening hyperparameter is set to 0.05, and the individual level envelope sample threshold screening hyperparameter is set to 0.7. The entire model is trained for 200 epochs, the initial learning rate is set to 0.01, and it is changed to 0.005 after 100 epochs. The above algorithm-independent parameter settings are based on the recognized experimental paradigm of related research.

[0231] Under the same experimental setting as above, five related models, STGAT, Transformer, Social-STGCNN

[23] , SGCN, and T-GNN, are compared with the MESC-HEDA model proposed in this paper. Tables 1 and 2 respectively give the ADE and FDE metric results of the model in 20 scene task combinations.

[0232] Table 1. ADE comparison between the MESC-HEDA model and the reference model

[0233]

[0234]

[0235] Table 2. FDE comparison between the MESC-HEDA model and the reference model

[0236]

[0237] As shown in Tables 1 and 2, the average values ​​of ADE and FDE indicators of the MESC-HEDA model proposed in the present invention are always better than those of the other five reference models. In general, the ADE index of the MESC-HEDA model is improved by 25.0% and 22.7% compared with the SGCN and T-GNN models, and the FDE index of the MESC-HEDA model is improved by 23.3% and 19.8% compared with the SGCN and T-GNN models. The MESC-HEDA model has the best ADE index in 13 of the 20 tasks, and the best FDE index in 12 of the 20 tasks. The possible reason is that the hierarchical envelope domain adaptation model obtains richer spatiotemporal feature representations of spatiotemporal sequence data at different levels by constructing multi-level envelope samples, and at the same time, it realizes domain distribution alignment by learning multi-level spatiotemporal feature information that can be transferred from the source domain to the target domain, further enhancing the model's multi-level cross-domain spatiotemporal feature transfer learning ability.

[0238] Therefore, most of the existing pedestrian trajectory prediction models do not consider cross-domain adaptation and do not solve the problem of pedestrian trajectory scene domain offset; a very small number of pedestrian trajectory prediction models based on domain adaptation only consider pedestrian local social level samples and their domain adaptation, and do not consider the sample construction and domain adaptation model construction of individual level and global level. The method of the present invention constructs multi-level hierarchical samples and integrates them with domain adaptation to construct a hierarchical domain adaptation trajectory prediction model, which effectively makes up for the defects of the above methods for the problem of large differences in trajectory data distribution between different pedestrian trajectory scenes. Specifically, the present invention proposes a pedestrian trajectory prediction method based on hierarchical envelope domain adaptation. The method constructs multi-level envelope samples and constructs a hierarchical envelope domain adaptation trajectory prediction model based on this, thereby fully constructing the spatiotemporal feature representation of pedestrian trajectory data at different levels, and effectively dealing with the problem of large distribution differences between trajectory domains of different scenes. Finally, multiple groups of experiments were conducted, and the experimental results show that the multi-level envelope samples and hierarchical envelope domain adaptation proposed by the present invention have better performance than other representative algorithms, and can effectively deal with the multi-level complexity of interactions between pedestrians and between pedestrians and the environment. In addition, the multi-level envelope sample learning module and the hierarchical domain adaptation module proposed in the present invention can be combined with other methods and have good reference value for related research.

[0239] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A pedestrian trajectory prediction method based on hierarchical envelope domain adaptation, It is characterized in that include: Obtain source domain original trajectory samples and target domain original trajectory samples; Construct a pedestrian trajectory prediction model based on hierarchical envelope domain adaptation, namely the MESC-HEDA model; The MESC-HEDA model is trained using the original trajectory samples in the source domain and the original trajectory samples in the target domain, and the pedestrian trajectory in the original trajectory samples in the target domain is predicted. The MESC-HEDA model includes: a two-level envelope sample construction module, a global level envelope sample construction module and a hierarchical envelope domain adaptation module, wherein the two-level envelope sample construction module includes a local level envelope sample construction submodule, an individual level envelope sample construction submodule, and a two-level envelope sample feature fusion submodule; the hierarchical envelope domain adaptation module includes a local level envelope domain adaptation submodule and a global level envelope domain adaptation submodule. The local level envelope sample construction submodule is used to generate local level envelope samples according to the pedestrian adjacency relationship at the sample space level, including local level envelope samples of the source domain and local level envelope samples of the target domain. The individual level envelope sample construction submodule is used to generate individual level envelope samples according to the pedestrian relationship at the individual level in the sample, including individual level envelope samples of the source domain and individual level envelope samples of the target domain. The two-level envelope sample feature fusion submodule is used to generate two-level envelope samples according to the local level envelope samples and the individual level envelope samples, including the bipolar envelope samples of the source domain and the bipolar envelope samples of the target domain. The global level envelope sample construction module is used to generate global level envelope samples according to the spatiotemporal feature distribution of the two-level envelope samples, including global level envelope samples of the source domain and global level envelope samples of the target domain. The local level envelope domain adaptation submodule is used to determine the local level domain distribution alignment loss using the attention mechanism based on the local level envelope samples of the source domain and the target domain. The global hierarchical envelope domain adaptation submodule is used to determine the global hierarchical domain distribution alignment loss based on the global hierarchical envelope samples of the source domain and the target domain.

2. The pedestrian trajectory prediction method based on hierarchical envelope domain adaptation according to claim 1, It is characterized in that According to the pedestrian adjacency relationship at the sample space level, local level envelope samples are generated, including: For any time t of all observed trajectories, that is, t∈{T 1 , ..., T obs }, T 1 , T 2 , ..., T obs Indicates the moment of observing the trajectory, execute S21-S27: S21: Get the original trajectory sequence samples of N pedestrians at time t, that is, the coordinates of N pedestrians at time t represents the coordinates of pedestrian i at time t, i∈{1, 2, ..., N}; S22: Processed as a graph representation G loc =(F loc , E loc ),in, Represents G loc The vertex set of Represents G loc The edge set of The edge formed by the coordinates of pedestrian i and pedestrian j is represented by the local level pedestrian adjacency relationship R loc and the local level adjacency matrix A loc express; S23: Determine the feature vector F of N pedestrians at time t loc ,in, represents the velocity vector of pedestrian i at time t; and They represent the x and y components of the coordinates of pedestrian i at time t respectively; and They represent the x and y components of the coordinates of pedestrian i at time t-1 respectively; S24: Mapping function F through local level envelope lhe , determine the local hierarchical pedestrian adjacency relationship R between any two pedestrians among all pedestrians loc , in, represents the speed of pedestrian j at time t, tr t j They represent the coordinates of pedestrian j at time t, and represents the x and y components of the coordinates of pedestrian j at time t; represents the Euclidean distance between pedestrians i and j after screening at time t; r 1 and r 2 Indicates the two distances set; represents the velocity vector of pedestrian i at time t The angle between the line connecting the current position of pedestrian j and the angle between Taking the cosine function we get It represents the comprehensive measure of the velocity angle between pedestrian i and pedestrian j at time t; S25: By introducing a threshold hyperparameter η∈[0, 1], R loc Perform local level envelope sample interaction strength screening to generate a local level mask M loc ,in, is the indicator function, when R loc When ≥η, output 1, otherwise output 0; S26: Element-wise multiplication operation on R loc Screening is performed and the unit matrix I is superimposed to generate the local hierarchical adjacency matrix A loc , that is, A loc =M loc ⊙R loc +I, ⊙ means element-by-element multiplication; S27: Zero-Softmax method is used to calculate the local hierarchical adjacency matrix A loc Normalize to generate a normalized local hierarchical adjacency matrix A′ loc , thus obtaining the local level envelope sample corresponding to time t [F loc , A′ loc ]; For all observed trajectories at the same time, loc and A′ loc Stacking is performed to obtain the local level envelope sample [F Loc , A Loc ], where F Loc ={F 1 , F 2 , ..., F obs }, F t represents the F corresponding to time t loc , A Loc ={A′ 1 , A′ 2 , ..., A′ obs },A′ t Denotes A′ corresponding to time t loc , t∈{T 1 , ..., T obs }.

3. The pedestrian trajectory prediction method based on hierarchical envelope domain adaptation according to claim 2, It is characterized in that According to the pedestrian relationships at the individual level in the sample, an individual level envelope sample is generated, including: For any pedestrian i among all pedestrians, that is, i∈{1, 2, ..., N}, where N represents the total number of pedestrians, execute S31-S37: S31: Get all observation trajectories at time T 1 , T 2 , ..., T obs Coordinates of insider i in, tr t i represents the coordinates of pedestrian i at time t, t∈{T 1 , ..., T obs }; S32: Processed as a graph representation G ind =(V ind , E ind ),in, G ind The vertex set of It's G ind The edge set of By the individual level pedestrian adjacency relation R ind and the individual level adjacency matrix A ind express; S33: Pedestrian i in time period T 1 to T obs The corresponding eigenvector F ind , represents the velocity vector of pedestrian i at time t; S34: Using the self-attention mechanism, determine the individual level pedestrian adjacency relationship R ind ,in, W Q and W K is a learnable parameter; Q ind and K ind Represent the query and key of the self-attention mechanism respectively; Indicates K ind Transpose; D represents the scale factor; F ind =(F ind +PE)W ind Represents a high-dimensional tensor with position encoding information; PE = {Frame t |t=T 1 , ..., T obs } represents a single pedestrian trajectory tr t i In time period T 1 to T obs Position encoding tensor within Frame t Indicates the time frame number at time t; W ind represents a learnable parameter; S35: By setting the threshold hyperparameter ψ∈[0, 1] ind Perform interactive screening of individual level envelope sample strength to generate individual level mask M ind ,in, is the indicator function, when R ind When ≥ψ, output 1, otherwise output 0; S36: Mask the individual level M ind And the pedestrian adjacency relation R at the individual level ind Fusion is performed to obtain the individual level adjacency matrix A ind , that is, A ind =M ind ⊙R ind , ⊙ represents element-by-element multiplication; S37: Use the Zero-Softmax method to perform the individual level adjacency matrix A ind Normalize and generate the normalized individual level adjacency matrix A′ ind , thus obtaining the individual level envelope sample corresponding to pedestrian i [F ind , A′ ind ]; F corresponding to all pedestrians loc and A′ loc Stacking is performed to obtain individual level envelope samples [F lnd , A Ind ], where F Ind ={F 1 , F 2 , ..., F N }, F i represents F corresponding to pedestrian i ind , A Ind ={A′ 1 , A′ 2 , ..., A′ N },A′ i Denotes A′ corresponding to pedestrian i ind , i∈{1, 2, ..., N}.

4. The pedestrian trajectory prediction method based on hierarchical envelope domain adaptation according to claim 3, It is characterized in that According to the local level envelope samples and the individual level envelope samples, a two-level envelope sample is generated, including: Perform graph convolution operation on the graph structure of the local level envelope sample to obtain the local level feature representation F′ of the local level envelope sample Loc , where F′ Loc =σ(A Loc F Loc W Loc ), σ represents the activation function ReLU(); W Loc represents a learnable parameter; Perform graph convolution operation on the graph structure of the individual level envelope sample to obtain the individual level feature representation F′ of the individual level envelope sample Ind , where F′ Ind =σ(A Ind F Ind W Ind ), σ represents the activation function ReLU(); W Ind represents a learnable parameter; The local level feature is represented as F′ Loc and individual level feature representation F′ Ind Perform splicing and fusion to obtain the feature representation F of the bipolar envelope sample tef ,in, Indicates splicing and fusion according to feature dimensions; represents the set of real numbers; L obs represents the total number of all observation moments; D represents the characteristic dimension of local level envelope samples and individual level envelope samples.

5. The pedestrian trajectory prediction method based on hierarchical envelope domain adaptation according to claim 4, It is characterized in that According to the temporal and spatial characteristic distribution of the two-level envelope samples, a global hierarchical envelope sample is generated, including: The characteristic representation of the bipolar envelope sample is F tef Input a multi-layer temporal convolutional network TCN with layers, and the output of the previous layer is used as the input of the next layer to obtain the predicted trajectory, where for the,th layer TCN: in, Represents the input of the,th layer TCN, W (l) is the learnable parameter of the l-th layer TCN, Represents the output of the ,th layer, that is, the predicted output of the multi-layer TCN; The output of the first layer The five parameters F of the binary Gaussian distribution of the predicted trajectory coordinates are determined through the linear layer final ,Right now Among them, Linear() represents linear layer processing, t∈[T obs+1 , T obs+2 , ..., T pre ], T obs+1 , T obs+2 , ..., T pre represents the time frame of the predicted trajectory, It represents the mean value parameter of the trajectory of pedestrian i at time t in the time frame of the predicted trajectory, which includes the mean value of the trajectory in the x and y directions, that is, and represents the standard deviation component of the trajectory of pedestrian i at time t, which includes the trajectory standard deviation in both the x and y directions, that is, and represents the correlation coefficient of the trajectory of pedestrian i at time t; Based on the mean parameter of each pedestrian's trajectory Calculate the linear speed of the pedestrian at time t Right now The prediction time period T obs+1 to T pre The linear velocity forms the vector U i ,Right now Among them, L pre Represents the forecast period T obs+1 to T pre The total number of predicted trajectory moments; All pedestrians’ U i Splice to get the global level envelope sample Right now 6. The pedestrian trajectory prediction method based on hierarchical envelope domain adaptation according to claim 4, It is characterized in that According to the local level envelope samples of the source domain and the target domain, the attention mechanism is used to determine the local level domain distribution alignment loss, including: Obtain local hierarchical feature representation of local hierarchical envelope samples of source and target domains and Among them, N s and N t Respectively represent the number of pedestrians in the local level envelope samples of the source domain and the target domain; The local hierarchical features of the source domain and the target domain are represented as F′ Loc(s) and F′ Loc(t) Decomposed according to pedestrians, that is in, For each local level envelope sample in the source domain, Will Flatten to D c The size of the feature vector, where D c =L obs ×D, Determine the interaction weights of different pedestrians in the local hierarchical envelope samples of the source domain Right now in, express The transpose of Determine the local hierarchical envelope of the source domain and the sum of the interactive effects between the pedestrian relationships J (s) ,Right now The J corresponding to all local level envelope samples in the source domain (s) Stacking is performed to obtain the local level pedestrian attention feature concentrated representation M of the source domain (s) ,Right now Among them, K (s) Represents the number of local hierarchical envelope samples in the source domain; For each local level envelope sample in the target domain, Will Flatten to D c The size of the feature vector, where D c =L obs ×D, Determine the interaction weights of different pedestrians in the local hierarchical envelope sample of the target domain Right now in, express The transpose of Determine the local hierarchical envelope of the target domain and the sum of the interactions between the samples J (t) ,Right now The J corresponding to all local level envelope samples in the target domain (t) Stacking is performed to obtain the local level pedestrian attention feature concentrated representation M of the target domain (t) ,Right now Among them, K (t) Represents the number of local hierarchical envelope samples in the target domain; The local hierarchical pedestrian attention features of the source domain and the target domain are centrally represented as M. (s) and M (t) , determine the local hierarchical domain distribution alignment loss L loc , that is, L loc =Dist 1 (M (s) , M (t) ), Dist 1 () represents the first distance function.

7. The pedestrian trajectory prediction method based on hierarchical envelope domain adaptation according to claim 6, It is characterized in that According to the global hierarchical envelope samples of the source domain and the target domain, the global hierarchical domain distribution alignment loss is determined, including: Get the global hierarchical envelope samples F of the source domain and the target domain glo(s) and F′ glo(t) ; Determine the linear velocity difference ratio between the source domain trajectory and the target domain trajectory represents the linear velocity of pedestrian i in the source domain at time t, represents the linear speed of pedestrian j in the target domain at time t, Ns represents the total number of pedestrians in the source domain in the training set, and Nt represents the total number of pedestrians in the target domain in the verification set; Determine the global hierarchical domain distribution alignment loss L glo , that is, L glo =Dist 2 (F glo(s) ,κ×F glo(t) ), where Dist 2 () represents the second distance function.

8. The pedestrian trajectory prediction method based on hierarchical envelope domain adaptation according to claim 7, It is characterized in that The objective function L when training the MESC-HEDA model using the original trajectory samples in the source domain and the original trajectory samples in the target domain is: L=L pre +λ 1 L loc +λ 2 L glo , Among them, L pre represents the prediction loss, λ 1 and λ 2 represents the weight, N represents the total number of pedestrians in the source domain, represents the probability density function, It represents the mean value parameter of the trajectory of pedestrian i at time t in the time frame of the predicted trajectory, which includes the mean value of the trajectory in the x and y directions, that is, and represents the standard deviation component of the trajectory of pedestrian i at time t, which includes the trajectory standard deviation in both the x and y directions, that is, and t represents the correlation coefficient of pedestrian i’s trajectory at time t.

Citation Information

Patent Citations

  • Street-crossing pedestrian trajectory prediction method based on SFM-LSTM neural network model

    CN114462667A

  • Pedestrian motion trail prediction method based on domain adaptation technology

    CN114723784A

  • Cross-domain migratable pedestrian trajectory prediction method based on depth map convolutional network

    CN115272395A

  • Model and method for multi-source domain adaptation by aligning partial features

    US20220138495A1

Cited By

  • Cross-regional trajectory prediction method and system based on class granularity space-time migration

    CN119272095A

  • A cross-regional trajectory prediction method and system based on class-granular spatiotemporal migration

    CN119272095B

  • Pedestrian trajectory prediction method and system based on dynamic envelope body

    CN120525912A

  • A pedestrian trajectory prediction method and system based on dynamic envelope

    CN120525912B