Pedestrian trajectory prediction method based on improved federated learning framework

Abstract

The invention provides an improved heterogeneous pedestrian trajectory prediction method under a federated learning framework. The method comprises the following steps: firstly, dividing a pedestrian data set through a random matrix to ensure that each client obtains heterogeneous data so as to increase the diversity of the data; next, local training is carried out on each client, the client can adopt a social space-time diagram convolutional neural network (Social-STGCNN) or a social attention network (STGAT), the adaptability of the model to local data is improved through fine adjustment of a private data set, and meanwhile similarity calculation is added into a loss function to maintain the stability of the model; and then, dynamically allocating weights by using a dynamic weight aggregation algorithm according to contribution indexes such as an average displacement error (ADE) and a final displacement error (FDE) so as to realize knowledge sharing of the heterogeneous model. Next, performing weighted average on client model parameters in the same group (namely, the same model type), updating a global model in the group, and transmitting the global model to each client in the group to ensure the consistency of initial parameters of the next round of training; and then knowledge distillation is carried out among different groups, and the consistency of prediction results when the heterogeneous model processes the same data is ensured by aligning model output. And finally, continuously optimizing the global model through multiple rounds of training until a preset training target is reached, and performing evaluation on the test set to improve the prediction precision and robustness of the model.

Description

Technical Field

[0001] The present invention relates to a pedestrian trajectory prediction system based on a federated learning framework, which predicts the future trajectory of pedestrians in a privacy-preserving manner through collaborative learning by multiple distributed participant devices. Background Art

[0002] Pedestrian trajectory prediction is a key technology in the fields of intelligent transportation systems, autonomous driving, and public safety. Traditional pedestrian trajectory prediction models usually rely on a large amount of centralized data for training, which may involve personal privacy and are prone to data leakage during transmission and storage. At the same time, as the amount of data increases, the computing and storage requirements of centralized training models also increase significantly.

[0003] Federated learning is an emerging distributed machine learning technology that can effectively protect data privacy and reduce data transmission costs by training models locally on multiple participants' devices and aggregating and updating global models. However, under the federated learning framework, participants' data may be heterogeneous (such as data collected in different environments) or have different quality (such as noisy data), which brings new challenges to model aggregation and updating.

[0004] The most similar prior art implementation to the present invention: Xiang Chen et al. proposed ATPFL to help users combine private and multi-source trajectory datasets to automatically design and train powerful trajectory prediction models. This is the first trajectory prediction work that simultaneously breaks data silos and technical limitations, allowing ordinary users to apply trajectory prediction technology more conveniently and effectively. The ATPFL framework is able to obtain a well-performing pedestrian trajectory prediction model under a federated learning framework, achieving better results than a manually designed pedestrian trajectory prediction model trained on a single-source dataset, which proves the effectiveness and importance of ATPFL.

[0005] Disadvantages of existing technologies: Existing federated learning frameworks usually rely on a unified model structure when dealing with pedestrian trajectory prediction, that is, the same pedestrian trajectory prediction model is used on all clients. Although this approach solves the problem of data silos to a certain extent, it has limited performance when facing dataset heterogeneity because it fails to fully consider the feature differences of data from different clients. In addition, existing frameworks usually distribute the dataset equally to each client without optimizing for data differences, which may lead to insufficient model generalization ability. Therefore, considering data heterogeneity and model heterogeneity during joint training is a challenging problem. Summary of the invention

[0006] Purpose of the invention: The high performance of pedestrian trajectory prediction models mainly depends on rich trajectory data from different scenarios. However, current research mainly adopts a centralized approach to manually train data from multiple scenarios. At the same time, there will be privacy issues between vehicles. Ultimately, the goal of the present invention is to provide a federated learning framework that can cope with data heterogeneity problems, so as to achieve more accurate pedestrian trajectory prediction in various complex scenarios, and improve the accuracy and robustness of pedestrian trajectory prediction through multi-party (autonomous driving vehicles) collaborative training models.

[0007] Technical solution: In order to achieve the above invention objectives, the technical solution of the present invention is as follows:

[0008] An improved heterogeneous pedestrian trajectory prediction under a federated learning framework includes the following steps:

[0009] S1. Use the randomness matrix to allocate scene data, and use the quantitative method based on the coefficient of variation to measure the heterogeneity of data allocation in federated learning. By adjusting the randomness factor, the heterogeneous changes under different allocation strategies can be observed to ensure that each client in federated learning obtains data with different characteristics, thereby enhancing the diversity of the model.

[0010] S2. Construct a participant model. Participant devices can use the same or different pedestrian trajectory prediction models as local models. It is mainly implemented by architectures such as Social Spatiotemporal Graph Convolutional Neural Network (Social-STGCNN) or Social Attention Network (STGAT). The model is fine-tuned using private datasets to enhance its adaptability to local data. Private data is obtained from the above heterogeneous data allocation.

[0011] S3. Using the dynamic weight aggregation algorithm, heterogeneous models dynamically assign weights based on the contribution of each client for knowledge sharing. The dynamic weight aggregation algorithm allows models with different architectures and performance to work together and effectively integrate the advantages of heterogeneous models through weight adjustment.

[0012] S4: Introduce the concept of a global model within the group, perform weighted averaging on the model parameters of the clients in the same group. The weights are dynamically assigned by the contributions in S3, and the updated global model parameters are passed to each client in the group to provide a unified initial model for the next round of local training.

[0013] S5. Implement knowledge distillation across groups by aggregating the models in each group into a global model and using the outputs of the local model, the global model in the group, and the global models of other groups to calculate the distillation loss during the knowledge distillation process. This can effectively share knowledge between heterogeneous models while maintaining the personalization of the local model. This solves the problem of model heterogeneity and establishes indirect knowledge transfer between different groups.

[0014] Furthermore, the step S1 comprises:

[0015] Partitioning of pedestrian trajectory dataset, including uniform distribution matrix P u , random proportion matrix P r , weighted combination matrix P c , and the final normalized distribution matrix P f :

[0016]

[0017] Where i represents the number of rows in the scene, j represents the number of columns in the client, n represents the number of clients, α is the randomness factor, and the final allocation matrix P f Normalize the weighted combination matrix so that the sum of the allocation ratios of each scene is 1, ensuring that the scene division is complete;

[0018] The data of different scenes obtained by each client are allocated according to the product of the allocation matrix and the amount of data in each scene, and the coefficient of variation CV is used to calculate the final To measure the heterogeneity between different clients, the formula is as follows:

[0019] μ is the mean of the data set, x i is the i-th element in the data, a is the size of the data set, and the standard deviation σ represents the degree of deviation of each data point from the mean. The formula is as follows The coefficient of variation (CV) is the ratio of the standard deviation to the mean. Represents the relative degree of discreteness, then according to the distribution matrix P f The row coefficient of variation can be obtained and the column coefficient of variation and the row coefficient of variation and and the column coefficient of variation and n represents the number of clients, and m represents the number of scenarios.

[0020] After distributing the data according to the product of the distribution matrix and the amount of data for each scene, the resulting data matrix A[i,j] is obtained, where the elements represent the amount of data that the j-th client gets for the i-th scene, and the sum of each column is It represents the total number of different scenarios obtained by the jth client, from which the data matrix columns and coefficient of variation are obtained Indicates the difference in the total amount of data from different clients. Finally, the product of these coefficients of variation is used as an evaluation indicator to measure the heterogeneity of the distribution CV f =CV r ×CV c ×CV sCoefficient of variation CV f The larger it is, the more random and heterogeneous the data distribution is.

[0021] Furthermore, in step S2:

[0022] The pedestrian trajectory prediction models used include Social-STGCNN and STGAT. The loss function of Social-STGCNN combines the probability density difference between the output of the time series graph convolutional neural network (STGCNN) model and the target value. The loss function calculates the difference between the predicted pedestrian trajectory and the actual trajectory, and calculates the loss through the bivariate normal distribution function under the Gaussian distribution assumption.

[0023] Specifically, the loss function of Social-STGCNN can be expressed as: V pred Represents the output of the model, which contains five parameters: μ x , μ y , σ x , σ y , and the correlation coefficient ρ. V trgt Represents the true value of the target.

[0024]

[0025] where μ x and μ y is the mean of the predicted trajectory, σ x and σ y is the standard deviation, controlling the uncertainty of the predicted trajectory, and ρ is the correlation in the x and y directions, controlling the curvature of the trajectory. By minimizing this loss function, Social-STGCNN optimizes its trajectory prediction so that its output is as close to the true pedestrian trajectory as possible.

[0026] Since the pedestrian trajectory prediction is under the federated learning framework, the local training is improved accordingly, so that a similarity calculation is added to the loss function. We divide different models for pedestrian trajectory prediction into different groups, one group uses STGAT, and the other group uses social-STGCNN. We design a global model G for each group. G is trained independently on local data by the client in the group, and then the model parameters are sent to the central server in the group for averaging, thereby updating the global model in the group and multiple local models trained in the tth round. i represents the local model of the client currently being processed. In each round of training, the local model It will calculate the global model G and the previous model Similarity:

[0027]

[0028] in, represents the similarity between the local model and the global model, Represents the local model M i The cosine similarity between and the global model G, where T is the temperature coefficient for scaling the similarity. Represents the local model and previous model The similarity between cos(M i ,P j ) represents the local model M i And the previous model P j The cosine similarity between . The similarity loss is calculated from this Through this process, the model is gradually optimized to reduce the prediction error while maintaining the similarity of the local model with the global model and the previous model. This approach ensures the stability and performance of the model when dealing with different data distributions.

[0029] Furthermore, the step S3 comprises:

[0030] S31: Calculate the test results of the client local training and the last test results recorded. The test results of this experiment are mainly evaluated by two indicators: ADE and FDE. ADE is the average displacement error between the predicted trajectory and the true trajectory, and FDE is the displacement error between the end point of the predicted trajectory and the end point of the true trajectory. The formula is as follows:

[0031]

[0032] where N is the number of pedestrians, T is the number of time steps of the trajectory (for ADE) or the last time step (for FDE), is the kth predicted position of the nth pedestrian at the tth time step, G n,T is the true position of the nnnth pedestrian at the tth time step, is the Euclidean distance (L2 norm) between the predicted position and the true position of the nth pedestrian at the tth time step, It means that among the K prediction samples, the one with the smallest error is selected. In ADE, all time steps t from 1 to T are considered, while in FDE, only the last time step T is considered.

[0033] S32: Detect local training test results ADE of each client i 、FDE i , the subscript i represents the i-th client. The heterogeneous models in federated learning need to measure the performance of each client model. v iis the contribution of the model, which is obtained by weighted average of the inverse of ADE and FDE. α is set to 0.5 here, indicating that ADE and FDE contribute equally to the final result. We convert smaller error values ​​into higher weights, that is, the better the model performs, the lower its error, and the higher the weight assigned.

[0034] S33: The heterogeneous model dynamically assigns weights to each client based on its contribution for knowledge sharing. In order to ensure that the weights of each client have a uniform proportion in the training of the global model, these weights need to be normalized. w i represents the weighted weight of the i-th client, N is the number of clients, and v i is the contribution of the i-th client. Although the models and data used by the clients may be different, the final global model can still learn more valuable features from the best performing client.

[0035] Furthermore, the step S4 comprises:

[0036] In a federated learning framework with heterogeneous clients, it is ensured that clients in the same group (i.e., the same type of model, such as STGAT or Social-STGCNN model) can improve the performance and consistency of the model within the group by sharing and aligning parameters, while maintaining the independence of the models between heterogeneous groups. The global model is generated by weighted averaging the model parameters of all clients in the group. Where K group is the number of clients in the group, is the weighted weight of each client, based on the contribution v in S3 i ,

[0037] Generated global model Will be sent back to each client in the group as the initial parameters for the next round of training This ensures the consistency and collaborative optimization of the models within the group, and guarantees knowledge sharing and model optimization within the group.

[0038] Furthermore, the step S5 comprises:

[0039] Since heterogeneous models may produce inconsistent outputs, we first need to map the outputs of these models to a unified dimension. In this project, the output of the Social-STGCNN model is a graph structure, while the output of the STGAT model is a time series structure. For consistency comparison and further calculation, we convert the graph structure output of Social-STGCNN to the same time series structure as STGAT. This ensures that the outputs of different models can be effectively compared and calculated on the same dimension.

[0040] Pedestrian trajectory prediction is a regression problem, in which we need to predict continuous values ​​(such as location coordinates), and mean square error is suitable for regression tasks. Under the federated learning framework, heterogeneous models can produce consistent prediction results when processing the same data. Using MSE can help align the outputs of these heterogeneous models so that their predictions under the same input are as close as possible. The formula is as follows: n is the number of samples, y i is the true value, is the predicted value.

[0041] Local models can learn knowledge from global models of other groups. Although their model architectures and data distributions are different, their prediction results can transfer knowledge through distillation.

[0042]

[0043] in is the prediction of the global model of other group k on the local data. Through this loss, the local model is not only aligned with the model within the group, but also benefits from the knowledge of other groups. Cumulative loss, for each model i, combined with the local model output loss loss1 and similarity loss loss2 in step S2, the overall loss function can be obtained:

[0044] Loss i =loss1+loss2+L group

[0045] For each model i, the model parameters are updated by backpropagation: θ i is the parameter of model i, η is the learning rate, is the gradient with respect to the parameter.

[0046] Through the global model within the group, the local model can quickly learn the knowledge of other models in the group. The model architecture and data between different groups may be quite different, but the cross-group global model can provide additional perspectives for the local model, helping it to better understand the diversity of data and improve generalization capabilities. This approach can well solve the problem of model heterogeneity and establish indirect knowledge transfer between different groups. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flowchart of improving pedestrian trajectory prediction under the federated learning framework.

[0048] Figure 2 This is a detailed flowchart of the steps for local training of the client model. DETAILED DESCRIPTION

[0049] The implementation method of the present invention will be further described below in conjunction with the accompanying drawings.

[0050] like Figure 1 As shown, the present invention designs a pedestrian trajectory prediction algorithm under an improved federated learning framework, comprising the following steps:

[0051] It is designed as 5 distributed clients, of which the first two are predicted by STGAT model, and the last three are predicted by Social-STGCNN model. The dataset takes trajectory data from two publicly available datasets ETH and UCY. The ETH dataset contains two real-life scenes, namely ETH (3666 data) and HOTEL (4946 data), each with 750 different pedestrians. The UCY dataset has 3 scenes: ZARA01 (4307 data), ZARA02 (7621 data) and UNIV (33994 data). For each client, we use 60%, 20%, and 20% of its local dataset as training set, validation set, and test set, respectively. The local round is set to 100.

[0052] Step 1: According to the allocation matrix P f [i,j] is used to distribute data. Since there are 5 clients, when the randomness factor is 0, it means uniform division. f Each element in [i,j] is When the random factor is 1, it means that the data heterogeneity is the strongest. The heterogeneity can be proved based on the corresponding coefficient of variation of the matrix. The experiment will be tested in three cases: the random factor is 0, 0.5, and 1.

[0053] Step 2: Multiply the allocation matrix obtained in step 1 by the total amount of data for each scene to obtain:

[0054]

[0055] Distribute to different clients to ensure that data for all scenarios is fully distributed and each client contains data for different scenarios, that is, the sum of each column of data.

[0056] Step 3: From the row and column coefficient of variation of the distribution matrix And the columns and final matrix A The corresponding coefficient of variation Get the product of the three CV f =CV r ×CV c ×CV s Used to measure the heterogeneity of the distribution.

[0057] Step 4: Deploy the pedestrian trajectory prediction model to the federated learning framework and determine whether to use the STGAT model or the social-STGCNN model for local training. The corresponding client is trained in the corresponding pedestrian trajectory prediction model.

[0058] Step 5: After training each model in step 4, calculate the test results of the client. ADE is the average displacement error between the predicted trajectory and the true trajectory. FDE is the displacement error between the predicted trajectory end point and the actual trajectory end point. According to the values ​​of these two indicators, record the contribution of each client:

[0059] Step 6: After a round of client training is completed, the contribution of each client is normalized. w i represents the weighted weight of the i-th client, N is the number of clients, and v i is the contribution of the ith client. The weighted weight is used for subsequent knowledge sharing dynamic weight allocation.

[0060] Step 7: Improve the local loss function during the training process of step 5, add a similarity calculation, design a global model G for different groups of models, which is obtained by averaging the parameters of the model members in the group and multiple local models trained in the tth round i represents the local model of the client currently being processed. Here we take the Social-STGCNN model as an example. The model with improved loss function is also modified in the Social-STGCNN model. The modifications in other models are similar, so I will not repeat them here.

[0061] In each round of training, the local model It will calculate the global model G within the group and the previous model Similarity:

[0062]

[0063] The similarity loss is calculated from this During the model training process, the model is gradually optimized, ensuring the stability and performance of the model in the face of heterogeneous data by reducing the prediction error while maintaining the similarity of the local model with the global model and the previous model.

[0064] Step 8: Since the outputs of pedestrian trajectory prediction models are diverse, STGAT uses time series output, while Social-STGCNN uses graph structure output. In order to share knowledge between the two outputs, it is first necessary to map their outputs to the same dimension. Here, the output of the latter is mapped into the same two-dimensional time series output as the former, so that the outputs of the two can be further processed.

[0065] Step 9: Pedestrian trajectory prediction is a regression problem, so here the output of the model in step 8 is constrained by the mean square error MSE, which can help align the outputs of these heterogeneous models so that their predictions under the same input are as close as possible. Local models can learn knowledge from global models of other groups. Although their model architectures and data distributions are different, their prediction results can be transferred through distillation.

[0066]

[0067] in is the prediction of the global model of other groups k on the local data. Through this loss, the local model is not only aligned with the model within the group, but also benefits from the knowledge of other groups.

[0068] Loss i =loss1+loss2+L group

[0069] Loss i Represents the output loss loss1 and similarity loss loss2 of the i-th client and the local model in step 7, and the overall loss function can be obtained.

[0070] Step 10: In the client-side heterogeneous federated learning framework, ensure that clients in the same group (i.e., the same type of model, such as STGAT or Social-STGCNN model) improve the performance and consistency of the model within the group by sharing and aligning parameters, while maintaining the model independence between heterogeneous groups. For each group (such as STGAT or Social-STGCNN), the central server needs to perform a weighted average of all client model parameters in the group to generate a global model within the group. Where K group is the number of clients in the group (e.g., the STGAT group has 2 clients, the Social-STGCNN group has 3 clients), is the weighted weight of each client, based on the contribution v of each client in step 6 i calculate: are the updated local model parameters after update.

[0071] Step 11: Global Model within Group After the update is completed, the server sends it to all clients in the group as the initial model parameters for the next round of training. i The initial model parameters in round t+1 are: This ensures that clients in the same group can start training with the same parameters in the next round of training, enhancing the similarity of models within the group and the effect of shared knowledge.

[0072] Step 12: After each round of training, check whether the model has reached the preset maximum number of training rounds. If not, continue with the next round of training and repeat steps 4 to 11 above. Otherwise, proceed to step 13.

[0073] Step 13: According to the global model obtained in step 11, test the test set and average the results as the experimental results ADE and FDE of the final model test.

Claims

1. An improved heterogeneous pedestrian trajectory prediction under a federated learning framework, comprising the following steps: S1. Use the randomness matrix to allocate scene data, and use the quantitative method based on the coefficient of variation to measure the heterogeneity of data allocation in federated learning. By adjusting the randomness factor, the heterogeneous changes under different allocation strategies can be observed to ensure that each client in federated learning obtains data with different characteristics, thereby enhancing the diversity of the model. S2. Construct a participant model. Participant devices can use the same or different pedestrian trajectory prediction models as local models. It is mainly implemented by architectures such as Social Spatiotemporal Graph Convolutional Neural Network (Social-STGCNN) or Social Attention Network (STGAT). The model is fine-tuned using private datasets to enhance its adaptability to local data. Private data is obtained from the above heterogeneous data allocation. S3. Using the dynamic weight aggregation algorithm, heterogeneous models dynamically assign weights based on the contribution of each client for knowledge sharing. The dynamic weight aggregation algorithm allows models with different architectures and performance to work together and effectively integrate the advantages of heterogeneous models through weight adjustment. S4: Introduce the concept of a global model within the group, perform weighted averaging on the model parameters of the clients in the same group. The weights are dynamically assigned by the contributions in S3, and the updated global model parameters are passed to each client in the group to provide a unified initial model for the next round of local training. S5. Implement knowledge distillation across groups by aggregating the models in each group into a global model and using the outputs of the local model, the global model in the group, and the global models of other groups to calculate the distillation loss during the knowledge distillation process. This can effectively share knowledge between heterogeneous models while maintaining the personalization of the local model. This solves the problem of model heterogeneity and establishes indirect knowledge transfer between different groups.

2. The method according to claim 1, characterized in that: The step S1 comprises: Partitioning of pedestrian trajectory dataset, including uniform distribution matrix P u , random proportion matrix P r , weighted combination matrix P c , and the final normalized distribution matrix P f : Where i represents the number of rows in the scene, j represents the number of columns in the client, n represents the number of clients, α is the randomness factor, and the final allocation matrix P f Normalize the weighted combination matrix so that the sum of the allocation ratios of each scene is 1, ensuring that the scene division is complete; The data of different scenes obtained by each client are allocated according to the product of the allocation matrix and the amount of data in each scene, and the coefficient of variation CV is used to calculate the final To measure the heterogeneity between different clients, the formula is as follows: μ is the mean of the data set, x i is the i-th element in the data, a is the size of the data set, and the standard deviation σ represents the degree of deviation of each data point from the mean. The formula is as follows The coefficient of variation (CV) is the ratio of the standard deviation to the mean. Represents the relative degree of discreteness, then according to the distribution matrix P f The row coefficient of variation can be obtained and the column coefficient of variation and the row coefficient of variation and and the column coefficient of variation and n represents the number of clients, and m represents the number of scenarios. After distributing the data according to the product of the distribution matrix and the amount of data for each scene, the resulting data matrix A[i,j] is obtained, where the elements represent the amount of data that the j-th client gets for the i-th scene, and the sum of each column is It represents the total number of different scenarios obtained by the jth client, from which the data matrix columns and coefficient of variation are obtained Indicates the difference in the total amount of data from different clients. Finally, the product of these coefficients of variation is used as an evaluation indicator to measure the heterogeneity of the distribution CV f =CV r ×CV c ×CV s Coefficient of variation CV f The larger it is, the more random and heterogeneous the data distribution is.

3. The method according to claim 1, characterized in that In step S2, the heterogeneous pedestrian trajectory model used is as follows: The pedestrian trajectory prediction models used include Social-STGCNN and STGAT. The loss function of Social-STGCNN combines the probability density difference between the output of the time series graph convolutional neural network (STGCNN) model and the target value. The loss function calculates the difference between the predicted pedestrian trajectory and the actual trajectory, and calculates the loss through the bivariate normal distribution function under the Gaussian distribution assumption. Specifically, the loss function of Social-STGCNN can be expressed as: V pred Represents the output of the model, which contains five parameters: μ x , μ y , σ x , σ y , and the correlation coefficient ρ. V trgt Represents the true value of the target. where μ x and μ y is the mean of the predicted trajectory, σ x and σ y is the standard deviation, controlling the uncertainty of the predicted trajectory, and ρ is the correlation in the x and y directions, controlling the curvature of the trajectory. By minimizing this loss function, Social-STGCNN optimizes its trajectory prediction so that its output is as close to the true pedestrian trajectory as possible.

4. The method according to claim 3, characterized in that Add the calculation of similarity loss function when performing local training on heterogeneous clients. Since the pedestrian trajectory prediction is under the federated learning framework, the local training is improved accordingly, so that a similarity calculation is added to the loss function. We divide different models for pedestrian trajectory prediction into different groups, one group uses STGAT, and the other group uses social-STGCNN. We design a global model G for each group. G is trained independently on local data by the client in the group, and then the model parameters are sent to the central server in the group for averaging, thereby updating the global model in the group and multiple local models trained in the tth round. i represents the local model of the client currently being processed. In each round of training, the local model It will calculate the global model G and the previous model Similarity: in, represents the similarity between the local model and the global model, Represents the local model M i The cosine similarity between and the global model G, where T is the temperature coefficient for scaling the similarity. Represents the local model and previous model The similarity between cos(M i ,P j ) represents the local model M i And the previous model P j The cosine similarity between . The similarity loss is calculated from this Through this process, the model is gradually optimized to reduce the prediction error while maintaining the similarity of the local model with the global model and the previous model. This approach ensures the stability and performance of the model when dealing with different data distributions.

5. The method according to claim 1, characterized in that In step S3, the method for calculating the dynamic weight is as follows: S31: Calculate the test results of the client local training and the last test results recorded. The test results of this experiment are mainly evaluated by two indicators: ADE and FDE. ADE is the average displacement error between the predicted trajectory and the true trajectory, and FDE is the displacement error between the end point of the predicted trajectory and the end point of the true trajectory. The formula is as follows: where N is the number of pedestrians, T is the number of time steps of the trajectory (for ADE) or the last time step (for FDE), is the kth predicted position of the nth pedestrian at the tth time step, G n,T is the true position of the nnnth pedestrian at the tth time step, is the Euclidean distance (L2 norm) between the predicted position and the true position of the nth pedestrian at the tth time step, It means that among the K prediction samples, the one with the smallest error is selected. In ADE, all time steps t from 1 to T are considered, while in FDE, only the last time step T is considered. S32: Detect local training test results ADE of each client i 、FDE i , the subscript i represents the i-th client. The heterogeneous models in federated learning need to measure the performance of each client model. v i is the contribution of the model, which is obtained by weighted average of the inverse of ADE and FDE. α is set to 0.5 here, indicating that ADE and FDE contribute equally to the final result. We convert smaller error values ​​into higher weights, that is, the better the model performs, the lower its error, and the higher the weight assigned. S33: The heterogeneous model dynamically assigns weights to each client based on its contribution for knowledge sharing. In order to ensure that the weights of each client have a uniform proportion in the training of the global model, these weights need to be normalized. w i represents the weighted weight of the i-th client, N is the number of clients, and v i is the contribution of the i-th client. Although the models and data used by the clients may be different, the final global model can still learn more valuable features from the best performing client.

6. The method according to claim 1, characterized in that In step S4, the method for constructing the global function within the group is as follows: In a federated learning framework with heterogeneous clients, it is ensured that clients in the same group (i.e., the same type of model, such as STGAT or Social-STGCNN model) can improve the performance and consistency of the model within the group by sharing and aligning parameters, while maintaining the independence of the models between heterogeneous groups. The global model is generated by weighted averaging the model parameters of all clients in the group. Where K group is the number of clients in the group, is the weighted weight of each client, based on the contribution v in S3 i , Generated global model Will be sent back to each client in the group as the initial parameters for the next round of training This ensures the consistency and collaborative optimization of the models within the group, and guarantees knowledge sharing and model optimization within the group.

7. The method according to claim 1, characterized in that In step S5, the knowledge distillation method of the cross-group global function is as follows: Since heterogeneous models may produce inconsistent outputs, we need to map the outputs of these models to a unified dimension. In this project, the output of the Social-STGCNN model is a graph structure, while the output of the STGAT model is a time series structure. For consistency comparison and further calculation, we convert the graph structure output of Social-STGCNN to the same time series structure as STGAT. This ensures that the outputs of different models can be effectively compared and calculated on the same dimension. Pedestrian trajectory prediction is a regression problem. We need to predict continuous values ​​(such as location coordinates). Mean square error is suitable for regression tasks. Under the federated learning framework, heterogeneous models can produce consistent prediction results when processing the same data. Using MSE can help align the outputs of these heterogeneous models so that their predictions under the same input are as close as possible. The formula is as follows: n is the number of samples, y i is the true value, is the predicted value. Local models can learn knowledge from global models of other groups. Although their model architectures and data distributions are different, their prediction results can transfer knowledge through distillation. in is the prediction of the global model of other group k on the local data. Through this loss, the local model is not only aligned with the model within the group, but also benefits from the knowledge of other groups. Cumulative loss, for each model i, combined with the local model output loss loss1 and similarity loss loss2 in step S2, the overall loss function can be obtained: Loss i =loss1+loss2+L group For each model i, the model parameters are updated by backpropagation: θ i is the parameter of model i, η is the learning rate, is the gradient with respect to the parameter. Through the global model within the group, the local model can quickly learn the knowledge of other models in the group. The model architecture and data between different groups may be quite different, but the cross-group global model can provide additional perspectives for the local model, helping it to better understand the diversity of data and improve generalization capabilities. This approach can well solve the problem of model heterogeneity and establish indirect knowledge transfer between different groups.

Images