A method for achieving heterogeneous perception and autonomous driving via federated learning without data distillation

Abstract

The present invention discloses a method for realizing heterogeneous perception autonomous driving through federated learning without data distillation. The method is based on a federated learning framework and combines knowledge distillation methods to unite autonomous driving vehicles into an alliance, and conducts training collaboration for the vehicle models in the alliance. Under the premise of ensuring the performance of the private model of each vehicle, the goal is to optimize the adaptability of the aggregated global model in federated learning to the private data in each vehicle, and class-based super knowledge is applied to knowledge distillation during training, thereby strengthening data sharing in autonomous driving vehicles and avoiding the generation of data privacy issues. The present invention can avoid the generation of data privacy leakage issues while achieving the optimization of individual models and global models, and can achieve better personal driving satisfaction.

Description

Technical Field

[0001] The present invention relates to the fields of federated learning, knowledge distillation, and autonomous driving, and specifically to a method for achieving heterogeneous perception autonomous driving through federated learning without data distillation. Background Art

[0002] In today's era, autonomous driving is a vibrant research field in both academia and industry. Sharing data between autonomous vehicles using cloud-based distributed learning is essential to continuously enhance predictive models through collective experience. However, there are huge challenges in balancing data sharing with the need to protect drivers' sensitive information and proprietary policies, especially when cooperating with other autonomous vehicles and preventing insurance privacy leaks. In this context, federated learning has emerged as a distributed machine learning framework. Through federated learning, multiple data owners can train machine learning models together, and the data remains local, and only the local model is transferred. As a result, data privacy is protected and network bandwidth for communication is saved. However, federated learning usually lacks robustness, and the performance of federated learning will seriously decline when user data is highly heterogeneously distributed.

[0003] In response to various heterogeneity issues (data, computing, communication, memory, power), the prior art also proposed personalized federated learning methods. Specifically, in this system, each client uses the information received from the server and trains its personalized model locally using customized objectives. So the client does not focus on global performance, but empirically evaluates the local performance of the model by running the local model on data whose distribution is similar to the distribution of local training data. However, since most personalized federated learning schemes still rely on gradient or model aggregation, they are very susceptible to "falling behind" (due to the poor performance of one client model, affecting other models), which slows down the training convergence process. Then some scholars proposed the method of federated prototype learning, which aims to solve the high communication cost and limitations of homogeneous models in federated learning. In this method, each client only sends the class prototype to the server - the average value of the sample representation of each class, rather than the model parameters. Although federated prototype learning improves local verification accuracy by utilizing aggregated class prototypes, it can hardly improve global performance. Inspired by the success of knowledge distillation, some researchers have inferred the soft prediction of samples as "knowledge" extracted from neural networks, and many federated learning methods have been proposed to improve the generalization ability of global models. However, most existing federated learning methods based on knowledge distillation require that public datasets be provided to all clients, which limits the feasibility of these methods in practical environments. To address this problem, some researchers have proposed using generative adversarial networks to synthesize public datasets, but this method requires additional high computing resources and the training of generative adversarial networks is very difficult.

[0004] Existing methods can neither take into account the performance of personalized local models and global models at the same time, nor can they perfectly solve the problem of public data sets. To this end, the present invention designs a method for realizing heterogeneous perception autonomous driving through federated learning without data distillation. This method can take into account the performance of personalized local models and global models at the same time, and does not need to rely on public data sets in federated learning. Therefore, this method improves the robustness of the model obtained by federated learning, enhances the driver's experience and safety in autonomous driving, and has practical significance and good application scenarios. Summary of the invention

[0005] Purpose of the invention: In view of the fact that the above-mentioned prior art cannot take into account the performance of personalized local models and global models at the same time, and cannot perfectly solve the problem of insufficient public data sets, the present invention provides a method for realizing heterogeneous perception autonomous driving through federated learning without data distillation.

[0006] Technical solution: A method for achieving heterogeneous perception autonomous driving through federated learning without data distillation. The method is based on a federated learning framework and combines knowledge distillation methods to unite autonomous driving vehicles into an alliance, and conducts training collaboration for the vehicle models in the alliance. On the premise of ensuring the performance of each vehicle's private model, the method aims to optimize the adaptability of the global model aggregated in federated learning to the private data in each vehicle, and applies category-based super knowledge to knowledge distillation during training, thereby strengthening data sharing in autonomous driving vehicles and avoiding the occurrence of data privacy issues.

[0007] The steps of the method are as follows:

[0008] S1. Building a federated learning alliance C represents each autonomous vehicle in the alliance, N represents the set of all vehicles in the alliance, and then the model of the autonomous vehicle is locally trained using the knowledge distillation method, so that after the local training is completed, the model of each vehicle and the category-based super knowledge are obtained;

[0009] S2. For the cars participating in the training, the local models and local super knowledge obtained by local training are uploaded to the FL server, and the FL server aggregates the global model and global super knowledge to obtain the global model and global super knowledge;

[0010] S3. After global aggregation is completed, the FL server sends the aggregated global model and global super knowledge to each autonomous driving car participating in the training in the alliance. The autonomous driving car uses the local model, local super knowledge and the sent global model and global super knowledge to perform knowledge distillation to obtain the updated local model and local super knowledge.

[0011] This training process continues until the set number of iterations or the global model converges.

[0012] Furthermore, in step S1, for the set of autonomous vehicles N, each autonomous vehicle i has a local dataset W i =(x i ,y i ), where x i Represents the original input data, y i Indicates the corresponding data label.

[0013] Step S1 defines a feature extractor and a classifier for local training, which are mathematically represented as follows:

[0014]

[0015] where x i represents the raw data of the autonomous vehicle i, and is a function of the feature extractor and classifier, and the model parameter is Φ i and Ω i ;h i is the intermediate layer representation extracted by the feature extractor, z i is a soft prediction represented by a classifier;

[0016] During local training, the average intermediate layer representation of class j in the local dataset of autonomous vehicle i is calculated as:

[0017]

[0018] in is the number of samples with label j in the dataset of autonomous driving i; Q(.,T) is the soft objective function with temperature T; and is the intermediate layer representation and i th k with label j for self-driving cars th Soft prediction of samples; average intermediate layer representation and the average soft prediction is the super knowledge of class j in client i, expressed as

[0019] If there are n classes, the complete category-based super-knowledge of autonomous vehicle i is

[0020] Furthermore, the aggregation method adopted in step S2 is the federated learning algorithm FedAvg, which is performed round by round in the training process and updates the parameter θ of the global model of the FL server to:

[0021]

[0022] where θ i It represents the parameters of the uploaded local model i that needs to be globally aggregated. i | represents the amount of data for local model i.

[0023] Furthermore, in step S2, after the FL server collects super knowledge from participating vehicles, the global super knowledge of class j in the global round t+1, K j,t+1 =(H j,t+1 ,Q j,t+1 ), which can be expressed as:

[0024]

[0025] in represents the number of samples in class j owned by client i, represents the local super-knowledge of class j about client i in the global tth round.

[0026] Beneficial effects: Compared with the prior art, the present invention proposes a method for achieving heterogeneous perception autonomous driving through federated learning without data distillation, which supports distributed updates of autonomous driving models between vehicles with heterogeneous data. This framework enables vehicles to simultaneously acquire a well-performing global model and a local model that adapts to the unique data distribution of each existing vehicle, which can be used to solve the cold start problem when a brand new vehicle joins the alliance. In addition, the present invention uses knowledge distillation to extract super knowledge based on local categories, and uses this super knowledge as a guide for local training to prevent dispersion in the direction of model optimization. At the same time, since the super knowledge is extracted based on categories, the entire training process does not require a public dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a schematic diagram of the overall structural framework of the method of the present invention;

[0028] Figure 2 It is a schematic diagram of the non-independent and identically distributed data structure of the autonomous driving car in the present invention;

[0029] Figure 3 This is a graph of experimental results comparing various algorithms when the number of autonomous vehicles in the alliance is different;

[0030] Figure 4 It is a graph of experimental results using different degrees of data heterogeneity. DETAILED DESCRIPTION

[0031] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and specific implementations. The following examples or drawings are used to illustrate the present invention, but do not limit the protection scope of the present invention.

[0032] A method for achieving heterogeneous perception autonomous driving through federated learning without data distillation. This method is based on the federated learning framework and combines the knowledge distillation method to unite autonomous driving cars into an alliance and conduct training collaboration for the car models in the alliance. Under the premise of ensuring the performance of each car's private model, the goal is to optimize the adaptability of the aggregated global model in federated learning to the private data in each car, and apply category-based super knowledge to knowledge distillation during training, thereby strengthening data sharing in autonomous driving cars and avoiding data privacy issues.

[0033] Specifically, the problems solved by this method are as follows:

[0034] (1) Under the premise of ensuring the performance of each car’s private model, optimize the adaptability of the aggregated global model in federated learning to the private data in each car;

[0035] (2) Introducing category-based super knowledge during training avoids the privacy leakage problem caused by private data.

[0036] See also Figure 1 , Figure 2 , a method for realizing heterogeneous perception autonomous driving through federated learning without data distillation, the method steps are as follows:

[0037] S1. First, Figure 1 As shown in the rounded rectangle on the left, building an alliance in federated learning C represents each autonomous vehicle in the alliance, and N represents the set of all vehicles in the alliance. The model of the autonomous vehicle is then trained locally using the knowledge distillation method, and the feature extractor and classifier are defined as follows:

[0038]

[0039] where x i represents the raw data of the autonomous vehicle i, and is a function of the feature extractor and classifier, and the model parameter is Φ i and Ω i ;h i is the intermediate layer representation extracted by the feature extractor, z i is a soft prediction represented by a classifier.

[0040] During local training, the average intermediate layer representation of class j in the local dataset of autonomous vehicle i is calculated as:

[0041]

[0042] in is the number of samples with label j in the dataset of autonomous driving i; Q(.,T) is the soft objective function with temperature T; and is the intermediate layer representation and i th k with label j for self-driving cars th Soft prediction of the sample. Average intermediate layer representation and the average soft prediction is the super knowledge of class j in client i, which is represented as If there are n classes, the complete super-knowledge of self-driving car i is

[0043] After the local training is completed, the model of each car and its super knowledge are obtained, such as Figure 1 As shown in ①.

[0044] S2, then for the cars participating in the training, upload the local models and local super knowledge obtained by local training to the FL server, such as Figure 1 As shown in ②. After the FL server collects super knowledge from participating vehicles, the global super knowledge of class j in the global t+1 round, K j,t+1 =(H j,t+1 ,Q j,t+1 ), which can be expressed as:

[0045]

[0046] in represents the number of samples in class j owned by client i, More importantly, this method emphasizes represents the local super-knowledge of class j about client i in the global tth round.

[0047] Thus we can obtain the global model and global super knowledge, such as Figure 1 As shown in ③.

[0048] S3. After the global aggregation is completed, the FL server sends the aggregated global model and global super knowledge to each autonomous driving car participating in the training in the alliance, such as Figure 1 As shown in ④, the autonomous driving car uses the local model, local super knowledge and the global model and global super knowledge sent down to perform knowledge distillation to obtain the updated local model and local super knowledge.

[0049] This training process continues until the set number of iterations or the global model converges.

[0050] A method for achieving heterogeneous perception autonomous driving through federated learning without data distillation. For a set of autonomous vehicles N, each autonomous vehicle i has a local dataset W. i ={(x i ,y i )}. i Represents the original input data, y i Indicates the corresponding data label. Figure 2 As shown, we can see that the data of different cars in the alliance are not independent and identically distributed. The data stored by cars traveling on highways, normal city roads, and rural areas may be very different.

[0051] See also Figure 3 , which is the result of the experiment conducted with different numbers of autonomous driving cars in the alliance under the same experimental conditions. It can be found that the accuracy of the global model obtained by the algorithm FKDHK proposed in the present invention is higher than that of the comparison algorithm in various cases, and the gap becomes more obvious as the number of autonomous driving cars in the alliance increases.

[0052] See also Figure 4 , this is the experimental result of using different data heterogeneity degrees under the same other conditions. It can be found that as the degree of non-independent and identically distributed data decreases, the performance of each algorithm will be improved, but the performance of the algorithm proposed in this invention is always the best.

[0053] Combined with Figure 1-4 The method of using the present invention is as follows: First, local training and knowledge extraction are performed for the car models in the alliance. Then, under the premise of ensuring the performance of the private model of each car, the local model and knowledge obtained by local training are uploaded to the server. After that, global model aggregation and global knowledge aggregation are performed. After the aggregation is completed, the obtained global model and global knowledge are distributed to each car participating in the training in the alliance.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for achieving heterogeneous perception autonomous driving through federated learning without data distillation, characterized in that: The method is based on the federated learning framework and combined with the knowledge distillation method to unite autonomous vehicles into an alliance, and conduct training collaboration for the vehicle models in the alliance. On the premise of ensuring the performance of each vehicle's private model, the method aims to optimize the adaptability of the aggregated global model in the federated learning to the private data in each vehicle, and applies category-based super knowledge to knowledge distillation during training, thereby strengthening data sharing in autonomous vehicles and avoiding the generation of data privacy issues. The steps of the method are as follows: S1. Building a federated learning alliance C represents each autonomous vehicle in the alliance, N represents the set of all vehicles in the alliance, and then the model of the autonomous vehicle is locally trained using the knowledge distillation method, so that after the local training is completed, the model of each vehicle and the category-based super knowledge are obtained; S2. For the autonomous driving vehicles participating in the training, the local models and local super knowledge obtained by the local training are uploaded to the FL server, and the FL server aggregates the global model and global super knowledge to obtain the global model and global super knowledge; S3. After global aggregation is completed, the FL server sends the aggregated global model and global super knowledge to each autonomous driving car participating in the training in the alliance. The autonomous driving car uses the local model, local super knowledge and the sent global model and global super knowledge to perform knowledge distillation to obtain the updated local model and local super knowledge. This training process continues until the set number of iterations or the global model converges.

2. The method for realizing heterogeneous perception autonomous driving through federated learning without data distillation according to claim 1, characterized in that: In step S1, for the set of autonomous vehicles N, each autonomous vehicle i has a local dataset W i =(x i ,y i ), where x i Represents the original input data, y i Indicates the corresponding data label.

3. The method for realizing heterogeneous perception autonomous driving through federated learning without data distillation according to claim 1, characterized in that: Step S1 defines a feature extractor and a classifier for local training, which are mathematically represented as follows: where x i represents the raw data of the autonomous vehicle i, and is a function of the feature extractor and classifier, and the model parameter is Φ i and Ω i ;h i is the intermediate layer representation extracted by the feature extractor, z i is a soft prediction represented by a classifier; During local training, the average intermediate layer representation of class j in the local dataset of autonomous vehicle i is calculated as: in is the number of samples with label j in the dataset of autonomous driving i; Q(.,T) is the soft objective function with temperature T; and is the intermediate layer representation and i th k with label j for self-driving cars th Soft prediction of samples; average intermediate layer representation and the average soft prediction is the super knowledge of class j in client i, expressed as If there are n classes, the complete category-based super-knowledge of autonomous vehicle i is 4. The method for realizing heterogeneous perception autonomous driving through federated learning without data distillation according to claim 1, characterized in that: The aggregation method used in step S2 is the federated learning algorithm FedAvg, which is performed round by round in the training process and updates the parameter θ of the global model of the FL server to: where θ i It represents the parameters of the uploaded local model i that needs to be globally aggregated. i | represents the amount of data for local model i.

5. The method for realizing heterogeneous perception autonomous driving through federated learning without data distillation according to claim 1, characterized in that: In step S2, after the FL server collects super knowledge from participating vehicles, the global super knowledge of class j in the global round t+1, K j,t+1 =(H j,t+1 ,Q j,t+1 ), which can be expressed as: in represents the number of samples in class j owned by client i, represents the local super-knowledge of class j about client i in the global tth round.

Images